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Abstract 


This  project  involves  the  investigation  of  best  practices  for  the  development  of  design  concepts  for 
a  visualization  aid,  specifically  an  alerting  system,  which  would  increase  the  RMP  operators’ 
awareness  and  understanding  of  maritime  anomalies  in  the  RMP  (e.g.  vessel  not  heading  to  port, 
grab  and  dash  fishing,  etc.).  Such  an  alerting  system,  however,  must  make  operators  aware  of 
anomalies  that  may  be  present  without  impacting  on  the  performance  of  their  primary  tasks. 

The  objectives  of  this  project  were  (i)  to  identify  and  analyse  available  literature  relevant  to  non- 
intrusive  alert  systems,  (ii)  develop  design  concepts  for  a  non-intrusive  alerting  interface  to  be  used 
in  GCCS-M  and  (iii)  obtain  feedback  from  Navy  Subject  Matter  Experts  (SMEs)  on  the  suitability 
of  the  design  options. 

The  results  of  the  literature  review  suggest  that  there  is  a  lack  of  a  unified  design  approach  and 
associated  recommendations  for  non-intrusive  alerting  contexts.  Furthermore,  there  was  no  single 
paper  that  definitively  addressed  the  issue  of  how  to  design  a  non-intrusive  alerting  system. 
However,  we  were  able  to  extract  relevant  concepts  from  the  literature  relating  to  alert/alarm 
design  in  general.  These  concepts,  combined  with  general  human  factors  principles,  provided 
direction  for  a  number  of  design  concepts  which  were  then  reviewed  and  evaluated  by  subject 
matter  experts. 

Future  design  efforts  should  work  toward  developing  an  alert  system  interface  design  in  accordance 
with  the  design  principles  listed  above,  once  these  design  requirements  have  been  validated. 
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Le  projet  comprend  1’ etude  des  meilleures  pratiques  applicables  a  la  definition  de  concepts  pour  un 
systeme  d’aide  a  la  visualisation,  en  l’occurrence  un  systeme  d’alerte,  qui  aiderait  les  operateurs  du 
TSM  a  mieux  connaitre  et  comprendre  les  anomalies  maritimes  indiquees  dans  le  TSM 
(deroutement  d’un  navire,  braconnage  maritime,  etc.).  Un  tel  systeme  d’alerte  doit  toutefois 
permettre  aux  operateurs  d’etre  informes  des  anomalies  eventuelles  sans  pour  autant  entraver 
l’execution  de  leurs  taches  principales. 

Le  projet  visait  les  objectifs  suivants  :  (i)  identifier  et  analyser  la  documentation  disponible  sur  les 
systemes  d’alerte  non  intrusive,  (ii)  elaborer  des  concepts  pour  une  interface  d’alerte  non  intrusive 
a  utiliser  dans  le  GCCS-M  et  (iii)  obtenir  la  retroaction  des  experts  de  la  Marine  sur  la  valeur  des 
options  de  conception. 

L’examen  de  la  documentation  a  revele  l’absence  d’une  approche  de  conception  unifiee  et  de 
recommandations  associees  dans  le  contexte  d’alertes  non  intrusives.  En  outre,  aucun  document 
n’offrait  de  solution  definitive  au  probleme  de  la  conception  d’un  systeme  d’alerte  non  intrusive. 
Toutefois,  on  a  pu  extraire  de  la  documentation  des  concepts  pertinents  pour  la  conception  de 
systemes  d’alerte  et  d’alarme  en  general.  Ces  concepts,  associes  a  des  principes  generaux  touchant 
les  facteurs  humains,  ont  fourni  des  orientations  pour  la  definition  d’un  certain  nombre  de  concepts 
qui  ont  ensuite  ete  examines  et  e  values. 

Les  recherches  futures  devraient  viser  a  definir  une  conception  d’ interface  de  systeme  d’alerte 
conformement  aux  principes  de  conception  presentes  ci-dessus,  une  fois  que  ces  exigences  de 
conception  auront  ete  validees. 
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Executive  Summary 


The  RMP,  one  of  the  primary  outputs  of  the  Regional  Joint  Operations  Centers  (RJOCs),  is 
essentially  a  map  of  the  Canadian  coastal  waters,  with  contacts,  typically  ships,  marked  on  the  map. 
Given  the  extensive  maritime  traffic  and  the  large  area  covered,  it  is  difficult  to  effectively  monitor 
the  RMP  to  maintain  a  good  understanding  of  the  current  situation,  including  anomalies  (e.g. 
sudden  increase  in  speed,  not  heading  to  port  of  call,  etc.).  Thus,  there  is  a  need  for  a  system  that 
could  perform  routine  checks  for  anomalous  data  in  the  background  and  then  make  the  information 
available  to  operators  in  a  format  that  makes  the  anomalies  readily  comprehensible  and  gives  rise 
to  rapid  situation  awareness.  The  crux  of  the  problem  is  how  to  implement  an  alerting  system  that 
would  make  operators  aware  of  anomalies  that  may  be  present  without  impacting  on  the 
performance  of  their  primary  tasks  (i.e.  non-intrusive  alert). 

The  objectives  of  this  project  were  (i)  to  identify  and  analyse  available  literature  relevant  to  non- 
intrusive  alert  systems,  (ii)  develop  design  concepts  for  a  non-intrusive  alerting  interface  to  be  used 
in  GCCS-M  and  (iii)  obtain  feedback  from  Navy  Subject  Matter  Experts  (SMEs)  on  the  suitability 
of  the  design  options. 

The  literature  review  attempted  to  uncover  human  factors  models,  design  guidelines,  design 
concepts  and  empirical  research  that  could  be  used  to  inform  the  design  of  the  functional  elements 
of  an  alerting  system,  including: 

•  Configuring  alert  parameters; 

•  Receiving  information  on  alert  states; 

•  Maintaining  awareness  and  performance  on  the  primary  task; 

•  Comprehending  the  alert  condition; 

•  Actioning  the  alert  condition;  and 

•  Managing  alerts. 

Results:  The  results  of  the  literature  review  suggest  that  there  is  a  lack  of  a  unified  design  approach 
and  associated  recommendations  for  non-intrusive  alerting  contexts.  Furthermore,  there  was  no 
single  paper  that  definitively  addressed  the  issue  of  how  to  design  a  non-intrusive  alerting  system. 
However,  relevant  concepts  from  the  literature  relating  to  alert/alarm  design  in  general  were 
extracted.  These  concepts,  combined  with  general  human  factors  principles,  provided  direction  for 
a  number  of  design  concepts  for  a  future,  non-intrusive  alerting  interface  for  maritime  anomalies. 

A  total  of  four  interface  design  concepts  were  developed  and  then  reviewed  and  evaluated  by  seven 
subject  matter  experts.  The  designs,  which  were  all  visual,  included  an  alert  indicator,  information 
related  to  an  incoming  alert  and  an  alert  management  window.  The  design  evaluation  identified  a 
need  to  scale  the  intrusiveness  of  alerts  (i.e.  high  priority  alerts  require  a  more  intrusive  alert  than 
those  of  low  priority). 

Significance:  Feedback  from  the  SMEs,  combined  with  consideration  of  general  human  factors 
principles,  resulted  in  a  list  of  design  requirements  for  the  best  way  to: 

•  Alert  RMP  operator  to  new  incoming  alert 

•  Provide  operator  with  awareness  of  the  number  of  active  alerts  in  the  system 

•  Provide  operator  with  information  specific  to  an  incoming  alert 

•  Provide  operator  with  information  on  all  active  alerts  in  the  system 
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•  Enable  operator  to  manage  (i.e.  action)  any  active  alerts  in  the  system 
Future  plans:  In  general,  future  research  and  development  efforts  should  focus  on: 

(i)  establishing  and  validating  detailed  design  requirements  for  an  alerting  system, 

(ii)  developing  an  alert  system  interface  design  in  accordance  with  the  design  principles 
presented  in  the  report, 

(iii)  experimentally  evaluating  alternate  design  options  for  an  anomaly  alert  system  in  the 
context  of  the  RMP  (i.e.  representative  of  user’s  work  environment  including  the  potential 
number  of  alerts), 

(iv)  basic  research  on  better  understanding  what  constitutes  intrusiveness  and  how  it 

relates  to  factors  such  as  attention  and  annoyance,  particularly  as  a  function  of 
alert  interruption  frequency. 


Page  iv 


Non-Intrusive  Alert  System 


Humansystems 


Sommaire 


Le  Tableau  de  la  situation  maritime  (TSM),  Tun  des  principaux  produits  des  centres  regionaux 
d'operations  interarmees  (CROI),  est  essentiellement  une  carte  des  eaux  coheres  canadiennes 
indiquant  des  contacts,  en  general  des  navires.  Etant  donne  Tampleur  du  trafic  maritime  et  la  vaste 
zone  couverte,  il  est  difficile  de  controler  efficacement  le  TSM  de  maniere  a  bien  comprendre  la 
situation  courante,  y  compris  les  anomalies  (augmentation  soudaine  de  vitesse,  deroutement  des 
navires,  etc.).  II  est  done  necessaire  de  disposer  d’un  systeme  capable  d’executer  des  verifications 
de  routine  pour  deceler  les  donnees  irregulieres  sur  Tarriere-plan,  puis  de  transmettre  ces 
informations  aux  operateurs  sous  une  forme  telle  que  les  anomalies  soient  immediatement 
comprehensibles  et  qu’on  puisse  prendre  rapidement  connaissance  de  la  situation.  Le  noeud  du 
probleme  consiste  a  trouver  un  moyen  de  mettre  en  oeuvre  un  systeme  d’alerte  qui  renseigne  les 
operateurs  sur  les  anomalies  possibles  sans  entraver  V execution  de  leurs  taches  principals  (alerte 
non  intrusive). 

Le  projet  visait  les  objectifs  suivants  :  (i)  identifier  et  analyser  la  documentation  disponible  sur  les 
systemes  d’alerte  non  intrusive,  (ii)  elaborer  des  concepts  pour  une  interface  d’alerte  non  intrusive 
a  utiliser  dans  le  GCCS-M  et  (iii)  obtenir  la  retroaction  des  experts  de  la  Marine  sur  la  valeur  des 
options  de  conception. 

L’examen  de  la  documentation  visait  a  decouvrir  des  modeles  de  facteurs  humains,  des 
orientations,  des  concepts  et  des  recherches  empiriques  qui  pourraient  etre  utilises  pour  guider  la 
conception  des  elements  fonctionnels  d’un  systeme  d’alerte,  ce  qui  comprend  : 

•  la  configuration  des  parametres  d’alerte; 

•  la  reception  d’ information  sur  les  etats  d’alerte; 

•  le  maintien  de  la  connaissance  de  la  situation  et  T  execution  des  taches  principales; 

•  la  comprehension  des  etats  d’alerte; 

•  Tactivation  de  l’etat  d’alerte; 

•  la  gestion  des  alertes. 

Resultats  :  L’examen  de  la  documentation  a  revele  Tabsence  d’une  approche  de  conception  unifiee 
et  de  recommandations  associees  dans  le  contexte  d’alertes  non  intrusives.  En  outre,  aucun 
document  n’offrait  de  solution  definitive  au  probleme  de  la  conception  d’un  systeme  d’alerte  non 
intrusive.  Toutefois,  la  documentation  a  permis  d’identifier  des  concepts  pertinents  pour  la 
conception  de  systemes  d’alerte  et  d’alarme  en  general.  Ces  concepts,  associes  a  des  principes 
generaux  touchant  les  facteurs  humains,  ont  foumi  des  orientations  pour  la  definition  d’un  certain 
nombre  de  concepts  pour  une  future  interface  d’alerte  non  intrusive  en  cas  d’anomalies  maritimes. 

En  tout,  quatre  concepts  d’interface  ont  ete  definis,  puis  examines  et  evalues  par  sept  experts.  Ces 
concepts,  tous  de  type  visuel,  comprenaient  les  suivants  :  indicateur  d’alerte,  information  liee  a 
une  nouvelle  alerte  et  fenetre  de  gestion  des  alertes.  L’ evaluation  de  la  conception  a  revele  qu’il 
fallait  prioriser  le  caractere  intrusif  des  alertes  (dans  les  cas  de  haute  priorite,  Talerte  doit  etre  plus 
intrusive). 

Portee  :  En  tenant  compte  de  la  retroaction  des  experts,  et  de  principes  generaux  touchant  les 
facteurs  humains,  on  a  etabli  une  liste  des  exigences  a  respecter  pour  la  conception  d’un  systeme 
qui  remplira  le  mieux  possible  les  fonctions  suivantes  : 

•  transmettre  les  nouvelles  alertes  a  Toperateur  du  TSM; 


Humansystems 


Non-Intmsive  Alert  System 


Page  v 


•  informer  l’operateur  du  nombre  d’alertes  actives  dans  le  systeme; 

•  foumir  a  l’operateur  1’ information  particuliere  a  une  nouvelle  alerte; 

•  fournir  a  l’operateur  1’ information  relative  a  toutes  les  alertes  actives  dans  le  systeme; 

•  permettre  a  l’operateur  de  gerer  (intervention)  toutes  les  alertes  actives  dans  le  systeme 

Recherches  futures  :  En  general,  les  futurs  travaux  de  recherche  et  de  developpement  devraient  se 
concentrer  sur  les  taches  suivantes  : 

(v)  etablir  et  valider  des  exigences  detaillees  pour  la  conception  d’un  systeme  d’ alerte; 

(vi)  definir  un  concept  d’ interface  de  systeme  d’ alerte  conformement  aux  principes  de 
conception  presentes  dans  le  rapport; 

(vii)  evaluer  experimentalement  differentes  options  pour  la  conception  d’un  systeme  d’ alerte 
en  cas  d’ anomalies  dans  le  contexte  du  TSM  (en  tenant  compte  de  l’environnement  de 
travail  de  l’utilisateur,  notamment  du  nombre  eventuel  d’alertes); 

(viii)mener  des  recherches  de  base  visant  a  mieux  comprendre  la  nature  de  l’intrusivite  et 
comment  celle-ci  se  rattache  a  des  facteurs  tels  que  l’attention  et  le  derangement, 
notamment  en  fonction  de  la  frequence  des  interruptions  en  cas  d’ alerte. 
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Introduction 


1. 


1.1  Background 

Defence  Research  and  Development  Canada  (DRDC)  has  an  ongoing  Applied  Research  Project  in 
the  Maritime  Intelligence,  Surveillance,  and  Reconnaissance  (MISR)  Thrust  on  information 
visualization  and  management  for  enhanced  domain  awareness  in  maritime  security.  This  is  a  4- 
year  Research  and  Development  (R&D)  project  with  the  goal  of  enhancing  the  “maritime  picture” 
through  improved  quality  of  information  and  novel,  adaptive  ways  of  visualizing  that  information. 
The  DRDC  team  wants  to  (i)  investigate  best  practices  for  the  design  of  new  visualization  aids  for 
operators  that  may  improve  their  understanding  of  issues  such  as  information  uncertainty  and  data 
anomalies,  and  (ii)  test  to  see  if  visualizing  this  information  can  help  improve  analysis  and  situation 
awareness  of  the  Recognized  Maritime  Picture  (RMP),  decision  making  based  on  the  RMP,  and  the 
RMP  operators’  efficiency  in  such  activities. 

1.1.1  Recognized  Maritime  Picture 

The  RMP  is  one  of  the  primary  outputs  of  the  Regional  Joint  Operations  Centers  (RJOCs).  In  its 
common  form,  the  RMP  is  a  map  of  the  Canadian  coastal  waters,  with  contacts,  typically  ships, 
marked  on  the  map.  Other  important  outputs  from  the  RJOCs  are  specific  reports  on  Vessels  of 
Interest  (VOI).  Although  produced  by  the  RJOCs,  the  RMP  and  VOI  reports  are  also  used  by  other 
government  agencies  (e.g.,  Department  of  Fisheries  and  Oceans,  the  Royal  Canadian  Mounted 
Police  and  Canadian  Navy  ships  at  sea  to  further  their  own  specific  interests).  Therefore,  it  is 
particularly  important  that  the  information  produced  in  the  RMP  and  VOI  reports  is  accurate  and 
reliable. 

The  term  “Recognized  Maritime  Picture”  implies  that  there  is  some  element  of  knowledge  of  the 
attributes  of  a  contact,  such  as  the  vessel’s  name,  hull  number  or  class  (e.g.,  tanker,  fishing  boat, 
warship).  A  typical  RMP  plot  would  show  hundreds  of  contact  symbols,  which  are  colour  coded 
using  standard  North  Atlantic  Treaty  Organization  (NATO)  conventions  for  identity  (friend, 
hostile,  neutral,  unknown)  and  may  also  contain  a  vector  arrow  to  indicate  last  known  direction  of 
movement.  Associated  with  each  plot  on  the  map  are  metadata,  which  are  shown  in  a  tabular  data 
display  containing  approximately  22  data  fields  of  information  concerning  the  contact.  In  addition, 
the  tabular  report  includes  detailed  information  on  the  reporting  source.  There  are  as  many  as  20-25 
reporting  sources  (Davenport,  Widdis  and  Rafuse,  2005)  that  could  potentially  contribute  to  a 
contact  report.  These  reporting  sources  range  from  highly  detailed  and  accurate  reports  from 
Provincial  Airlines  (PAL)  overflights,  Canadian  Coastguard  and  Canadian  Navy  ships  at  sea  as 
well  as  detailed  information  on  a  ship’s  name  and  position  from  ships  carrying  Automatic 
Identification  System  (AIS)  transponders  to  single  contact  reports  from  radar  sources  such  as  High 
Frequency  Surface  Wave  Radar  (HFSWR)  and  Electronic  Intelligence  (ELINT). 

The  many  sensor  and  reporting  sources  that  contribute  to  the  RMP,  therefore,  push  large  volumes 
of  data  into  associated  databases.  The  RMP  is  constantly  being  updated,  both  manually  by 
operators  and  automatically.  Given  the  extensive  maritime  traffic  and  the  large  area  covered,  it  is 
difficult  to  effectively  monitor  the  RMP  to  maintain  a  good  understanding  of  the  current  situation. 
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1.1 .2  Maritime  anomalies 

Anomalies  are  defined  as  deviations  from  the  norm  (Riveiro,  Falkman,  &  Ziemke,  2008)  or  targets 
that  are  not  easily  classified  (Roy,  2008).  The  more  common  types  of  RMP  Anomalies  include 
attribute,  movement  and  VOI-related  anomalies. 

Attribute  related  anomalies:  In  the  Concept  of  Operations  (CONOPS)  for  producing  the  RMP,  a 
track  refers  to  one  ore  more  contact  reports  linked  together  to  describe  any  detected  discrete 
airborne,  surface  or  subsurface  object.  A  track  has  many  attributes,  any  of  which  can  be  missing, 
incorrect  (e.g.,  misspelled)  or  inconsistent  (changed  by  different  operators,  or  characterized  by 
different  standards/nomenclature/schemas  as  tracks  pass  through  various  RMP  management  sites). 
Such  problems  cause  ambiguous  tracks  in  some  cases  or  just  bad  information  on  a  contact  in 
others.  At  the  moment,  the  most  common  alerting  method  for  attribute  mismatch  is  the  production 
of  an  ambiguous  track  by  Global  Command  and  Control  System-Maritime  (GCCS-M),  a  de  facto 
form  of  alerting  that  there  is  a  problem  with  a  track,  which  the  operator  then  must  resolve. 

Movement  related  anomalies  include: 

•  Unexpected  or  illogical  course  and  speed  changes 

•  Failure  to  reach  a  reported  estimated  time  of  arrival  (ETA)  at  a  specified  point 

•  Loss  of  reporting 

•  Movement  that  suggests  activity  of  concern,  such  as  very  close  proximity  to  other  vessels 
in  open  waters 

•  Significant  variance  in  voyage/route  compared  to  historical  trace  of  other  recent  voyages 

VOI  related  anomalies:  The  Maritime  Forces  Atlantic  (MARLANT)  “Alert  Table”  is  a  list  that 
extracts  key  fields  (usually  vessel  name)  from  RMP  messages  if  the  vessel  has  been  previously 
entered  on  the  list  (mostly  VOIs).  This  “alert”  prompts  operators  to  contact  whoever  is  listed  as  the 
interested  agency  for  the  VOI.1 

1 .1 .3  Detection  of  anomalies 

Detection  of  anomalies  requires  identification  of  whether  or  not  specific  behaviours  are  abnormal. 
A  model  of  this  process  has  been  proposed  by  Riveiro  and  colleagues  (2008).  According  to  the 
model,  anomaly  detection  is  a  complex  process,  requiring  information  acquisition  (search  and 
detection),  analysis  and  the  ability  to  integrate  events  within  the  operator’s  situation  awareness  and 
mental  model  of  the  operational  environment.  In  the  present  case,  this  would  be  bounded  by  the 
information  provided  by  existing  operational  systems  (e.g.,  GCCS-M,  the  Contact  History 
Database  (CHDB)). 

Operators  who  may  be  required  to  perform  anomaly  detection  in  conjunction  with  their 
visualization  aids  face  a  number  of  different  challenges.  Rhodes  (2007)  points  out  two  specific 
problems  associated  with  the  detection  and  prediction  of  anomalous  behaviour.  First,  normalcy  is 
dependent  on  the  context  in  question.  It  is  nearly  impossible  to  create  a  system  that  recognizes 
every  type  of  normal  and  abnormal  behaviour.  Thus,  a  system  that  is  always  learning  and  adapting 
is  needed  to  deal  with  such  complexity.  However,  such  systems  need  to  function  reliably  without 
requiring  a  high  level  of  operator  involvement. 


1  For  many  operators,  this  would  be  their  only  concept  of  an  alerting  method  together  with  the  method  used 
to  represent  ambiguous  track  generation  in  GCCS-M.) 
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In  addition  to  anomaly  identification,  RJOC  operational  staff  has  extremely  demanding  work 
responsibilities  (Davenport  et  al.,  2005;  Matthews,  Bruyn,  Keeble  &  Rafuse,  2004)  that  effectively 
preclude  them  from  spending  time  doing  detective  work  at  uncovering  data  anomalies.  None  of  the 
current  job  functions  or  tasks  performed  at  the  RJOCs  includes  any  effort  directed  specifically  at 
anomaly  detection  or  analysis  with  the  possible  exception  of  VOIs  (Matthews  et  ah,  2004).  Thus, 
there  is  a  need  for  a  system  that  could  perform  routine  checks  for  anomalous  data  in  the 
background  and  then  make  the  information  available  to  operators  in  a  format  that  makes  the 
anomalies  readily  comprehensible  and  gives  rise  to  rapid  situation  awareness.  The  crux  of  the 
problem  is  how  to  implement  an  alerting  system  that  would  make  operators  aware  of  anomalies 
that  may  be  present  without  impacting  on  the  performance  of  their  primary  tasks. 

1 .2  Scope  and  objectives 

The  objectives  of  the  project  as  a  whole  were  (i)  to  identify  and  analyse  available  literature  relevant 
to  non-intrusive  alert  systems,  (ii)  develop  design  concepts  for  a  non-intrusive  alerting  interface  to 
be  used  in  GCCS-M  and  (iii)  obtain  feedback  from  Navy  Subject  Matter  Experts  (SMEs)  on  the 
suitability  of  the  design  options. 

The  literature  review  provided  direction  for  a  number  of  design  concepts  for  a  future,  non-intrusive 
alerting  interface.  This  alerting  system  interface  must  serve  the  information  requirements  of  operators 
and  improve  their  situation  awareness  of  anomalies  and  their  meaning  for  the  RMP.  Because  operators 
must  multi-task  and  there  exists  a  potential  for  a  large  number  of  data  anomalies  to  occur,  the  major 
constraint  for  the  interface  design  is  that  it  be  “non-intrusive”.  A  total  of  five  design  concepts  were 
developed,  four  of  which  were  demonstrated  for  naval  SMEs  in  order  to  gather  user  feedback. 

1 .3  Non-intrusive  alerting 

Non-intrusive  alerting  is  not  a  clearly  defined  concept  that  can  be  expressed  in  absolute  terms,  but 
is  highly  context  dependent.  Essentially,  the  concept  of  non-intrusiveness  conveys  the  notion  of 
advising  an  operator  (creating  awareness)  that  an  alert  condition  has  occurred  in  a  manner  that  does 
not  disrupt  ongoing  task  performance.  It  should  be  noted  that  a  preliminary  search  indicated  that 
there  is  limited  literature  available  that  deals  specifically  with  the  issue  of  the  intrusiveness  of  an 
alert,  and  how  to  scale  this  intrusiveness  to  a  particular  set  of  operational  circumstances.  Therefore, 
the  review  and  analysis  of  much  of  the  following  literature  was  to  determine  what  useful 
information  could  be  extracted  from  the  literature  on  more  generic  aspects  of  alarm  and  alerting 
systems  that  would  be  applicable  or  extensible. 

1 .4  Outline  of  report 

Sections  2  and  3  describe  the  basic  ontology  of  an  alert  system  as  well  as  a  working  definition  of 
non-intrusiveness.  These  sections  are  based  on  the  knowledge  that  was  gained  by  reviewing  the 
literature  and  provide  a  framework  for  discussing  the  results  of  the  literature  review.  Section  4 
presents  the  method,  results  and  conclusion  of  the  literature  review.  In  section  5,  we  provide 
examples  of  design  concepts  for  elements  of  a  non-intrusive  alerting  system  for  implementation  in 
a  GCCS-M  environment.  This  is  supplemented  by  a  separate  PowerPoint  interactive  demonstration 
that  allows  a  cognitive  walkthrough  of  the  design  alternatives.  Section  6  describes  the  review  and 
assessment  of  the  design  concepts  by  SMEs  from  RJOC-Atlantic.  The  final  section  of  the  report 
gives  the  overall  conclusions  and  recommendations. 
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2.  The  Basic  Ontology  of  an  Alert  system 


Prior  to  reviewing  the  results  of  the  literature  search,  we  provide  in  this  section  an  outline  of  the 
functional  elements  of  an  alerting  system  which  we  believe  brings  a  useful  and  coherent  structure 
or  framework  for  the  diverse  papers  reviewed.  These  ideas,  in  themselves,  stem  from  insights 
gained  in  the  literature  review.  Our  objective  is  to  define  the  functional  elements  an  alerting  system 
and  how  they  relate  to  each  other. 

In  addition,  certain  constraints  and  assumptions  have  guided  this  analysis  concerning  the  role  of  the 
operator  and  the  team,  as  follows: 

•  The  operator  has  a  primary  task  or  tasks  to  fulfil  which  are  NOT  the  monitoring  or 
management  of  alerts; 

•  The  operator  does  not  have  any  other  team  members  who  will  monitor  or  manage  alerts; 
and 

•  The  goal  of  the  operator  is  to  deal  with  alerts  as  efficiently  as  possible  and  to  return 
promptly  to  the  primary  task. 

The  functional  elements  of  an  alerting  system  serve  several  different  tasks  that  a  user  may  be 
required  to  perform  in  the  set-up,  operation  and  maintenance  of  an  alerting  system.  These  have 
been  adapted,  and  expanded  upon,  from  McCrickard,  Chewar,  Somervell  and  Ndiwalana  (2003a) 
and  are  outlined  below: 

•  Configuring  alert  parameters; 

•  Receiving  information  on  alert  states; 

•  Maintaining  awareness  and  performance  on  the  primary  task; 

•  Comprehending  the  alert  condition; 

•  Actioning  the  alert  condition;  and 

•  Managing  the  accumulated  alert  information. 

The  elements  of  the  ontology  are  shown  in  Figure  1  and  are  outlined  below.  Explanation  of  the 
colour  and  line  coding  is  provided  in  section  2.7. 


2  The  essential  meaning  of  an  ontology  is  that  it  is  a  model  for  describing  the  world  that  consists  of  a  set  of 
types,  properties,  and  relationship  types.  Exactly  what  is  provided  around  these  varies,  but  they  are  the 
essentials  of  an  ontology  (Gmber,  1995). 
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Figure  1:  Relationship  of  literature  to  the  alert  ontology 


2.1  Configure  alert  parameters 

In  an  operational  environment  such  as  the  RJOC,  the  potential  exists  for  many  data  anomalies  to 
trigger  alerts.  Some  anomalies  will  be  considered  to  be  high  priority,  others  low.  In  addition, 
different  circumstances  may  require  different  rules  to  be  put  into  place  concerning  when  an  alert  is 
brought  to  the  attention  of  the  operator.  Therefore,  a  critical  component  of  the  system  is  a  function 
that  allows  managers  and  operators  to  define  the  criteria  for  an  alert  and  to  assign  different  alerting 
priorities  to  these  events.  The  interface  for  this  function  should  allow  operators  to  simply  create 
rules  by  picking  among  menu  choices,  for  example: 


“in  area  (set  co-ordinates),  if  vessels  of  (type)  have 
speed  =  (value)  then  alert  with  prioritv  (x)” 


For  the  purposes  of  the  present  project,  this  aspect  of  an  alerting  system  was  considered  out  of 
scope  to  be  examined  in  detail;  therefore  no  specific  literature  search  was  conducted  on  this  topic, 
although  one  useful  paper  that  emerged  from  the  more  general  search  was  reviewed.  However,  we 
believe  that  the  ability  to  set  alert  parameters  that  are  suitably  configured  to  an  operational  context 
represents  an  important  aspect  of  reducing  nuisance  alarms.  In  addition,  allowing  operators  to  set 
categories  of  alarm  priority  and  matching  these  categories  to  the  way  the  alarm  state  is  conveyed, 
represents  an  important  aspect  that  is  highly  relevant  to  the  intrusiveness  of  the  alarm. 
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2.2  Receive  information  about  the  alert  state 


This  is  the  key  component  of  the  system  and  incorporates  the  means  by  which  an  alert  state  will  be 
brought  to  the  attention  of  the  operator.  This  function  is  concerned  with  just  the  notification  and 
does  not  provide  details  concerning  the  nature  of  the  condition  that  gave  rise  to  the  alert.  However, 
the  alert  notification  should  provide  information  about  the  alert  priority.  The  design  of  the  alert 
indicator  is  the  principle  focus  of  this  project 

2.3  Comprehend  the  alert  condition 

Having  been  notified  of  an  alert,  at  some  point  the  operator  will  decide  to  process  the  alert  and 
needs  further  information  about  the  condition(s)  that  resulted  in  the  alert.  This  information  needs  to 
be  provided  in  a  succinct  manner  that  allows  the  operator  to  readily  comprehend  the  type  of  alert. 

In  the  case  of  the  RJOC,  the  interface  should  allow  the  operator  to  rapidly  identify  the  vessel 
involved  and  the  area  of  the  RMP  concerned.  This  functionality  is  a  secondary  point  of  focus  for 
this  project. 

2.4  Action  the  alert  condition 

This  is  a  function  that  allows  the  operator  to  investigate  the  alert  and  take  whatever  action  is 
required.  This  functionality  may  be  integrated  into  the  interface  for  an  alerting  system  or  may 
involve  other  work  functions  independent  of  the  alert  system.  It  could  be  considered  that  this 
function  is  not  really  part  of  an  alerting  system  at  all;  however,  at  some  point  the  operator  will 
return  to  the  primary  task,  therefore  the  interface  will  need  to  provide  suitable  functionality  to 
support  this  transition.  While  the  actioning  of  an  alert  was  out  of  scope  for  this  project,  the 
regaining  of  situation  awareness  of  the  primary  task  was  considered  important  and  is  dealt  with 
more  extensively  under  the  function  of  Maintain  Primary  Task. 

2.5  Manage  alerts 

As  alerts  accumulate  in  the  operating  environment,  a  new  task  relating  to  their  management  arises 
for  the  operator  and  possibly  system  manager.  The  database  of  these  alerts  will  need  to  be 
structured  and  formatted  in  such  a  manner  that  it  permits  the  operator  to  quickly  determine,  for 
example,  how  many  alerts  are  in  the  system,  what  are  there  priorities  and  when  they  occurred. 
Simple  tools  for  the  sorting  and  deletion  of  records  will  also  be  required.  This  aspect  of  an  alerting 
system  was  not  the  primary  focus  of  the  literature  review,  however,  some  basic  functional  elements 
for  the  interface  for  such  a  system  have  been  developed  and  are  shown  later  in  the  report  in  the 
section  on  design  concepts.  This  inclusion  was  thought  to  be  necessary  because  effective  design  of 
the  process  for  alert  management  may  reduce  operator  worker  load  for  this  activity  and  indirectly 
mitigate  the  information  processing  costs  associated  with  processing  the  alarm,  thereby  potentially 
reducing  the  overall  alarm  intrusiveness  into  the  normal  workflow. 

2.6  Maintain  primary  task 

The  operator’s  need  to  maintain  focus  on  his/her  primary  task  responsibilities  is  not  really  part  of 
an  alerting  system;  however  it  does  have  implication  on  the  relationship  between  this  primary  task 
and  various  aspects  of  the  tasks  performed  in  responding  to  and  managing  alerts.  Some 
considerations  are  when  to  present  information  about  an  alert  state,  i.e.,  when  is  the  best  time  to 
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interrupt  the  primary  task?  Should  warnings  be  provided  of  impending  alerts  to  allow  the  operator 
to  better  prepare  and  not  suddenly  lose  primary  task  focus  by  a  surprise  alert?  A  further 
consideration  is  how  to  enable  the  operator  to  regain  primary  task  situation  awareness  after  dealing 
with  an  alert.  The  focus  of  the  present  project  is  only  on  the  last  of  these  three  questions. 

2.7  Organisation  of  the  literature  and  the  ontology 

In  order  to  structure  the  literature  review  in  terms  of  its  relevance  to  the  ontology,  a  classification 
system  was  applied  that  broke  the  literature  down  into  four  main  categories: 

1 .  Conceptual  models 

2.  Generic  alert  system  design  guidelines 

3.  Specific  design  concepts  for  alerts 

4.  Relevant  empirical  research  on  alert  systems 

In  Figure  1,  we  show  the  specific  links  between  these  categories  and  those  elements  of  an  alerting 
system  that  were  the  focus  of  the  literature  search.  The  primary  areas  of  interest  are  shown  in 
green  boxes  and  are  linked  to  the  literature  review  by  solid  lines.  The  secondary  areas  are  shown  in 
light  yellow  and  the  links  are  shown  by  broken  lines.  Note  that  there  is  no  link  to  the  “tools  to 
service  alerts”  function,  since  this  was  completely  beyond  the  scope  of  this  project. 
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3.  Towards  a  Working  Definition  of  “Non- 

intrusiveness” 

3.1  Defining  non-intrusiveness 

Prior  to  considering  any  design  options  for  a  non-intrusive  alert  system  and  in  order  to  better 
understand  the  literature  review,  we  believe  that  it  is  necessary  to  clarify  what  is  meant  by 
“intrusive”  and  to  develop  a  working  operational  definition  that  will  guide  the  design  process, 
which  is  described  later  in  the  report.  As  a  starting  point,  we  make  the  assumption  that  the 
intrusiveness  of  an  alert  or  alarm  can  be  defined  as  an  event  which,  either  immediately,  or  over 
time,  reduces  the  ability,  or  potential  capacity,  to  perform  a  primary  task. 

With  this  definition  in  mind,  a  central  question  is  then  what  psychological  models  are  relevant  to 
intrusiveness  and  how  do  they  predict  what  would  constitute  an  intrusive  or  non-intrusive  alert. 
There  appear  to  be  two  classes  of  models  that  are  relevant  to  this  issue.  The  first  class  of  models 
deals  with  attention  and  resource  sharing  and  the  second  class  concerns  models  of  annoyance. 
Attentional  or  resource  sharing  models  (e.g.,  Wickens,  1984)  predict  interference  in  cognitive 
processing  of  a  primary  task  when  an  alert  competes  for  common  resources.  On  the  other  hand, 
annoyance  models  would  suggest  that  it  is  some  quality  of  the  alarm  (e.g.,  for  auditory  alarms, 
intensity,  frequency  components,  duration  and  repetition  cycle)  that  produces  a  psychological  state 
of  annoyance  that  becomes  so  compelling  that  attention  can  no  longer  be  devoted  to  the  primary 
task  (e.g.,  De-Muer,  Botteldooren,  Coensel,  Berglund,  Nilsson  &  Lercher,  2005). 

3.1.1  Predictions  of  attentional/resource  models 

According  to  attention  and  resource  sharing  models,  an  alarm  would  be  considered  intrusive  if  it 
directly  affects  the  processing  of  the  primary  task  by  sensory  interference.  For  a  visual  monitoring 
task  this  might  be  a  visual  alert  that  occurs  at  the  primary  point  of  visual  focus  thereby  directly 
interfering  with  the  perception  of  visual  information  required  to  monitor  the  display.  Other 
examples  of  this  might  be  the  whole  display  flashing  on  and  off,  a  visual  message  that  pops  up  in 
the  middle  of  the  screen  or  a  siren  that  occurs  over  an  auditory  communication  network.  A  second 
method  of  interference  suggested  by  the  attentional/resource  models  would  be  an  alarm  that 
competes  in  a  compelling  way  for  attentional  resources,  but  does  not  in  itself  directly  interfere  with 
the  perceptual  or  cognitive  processing  of  the  primary  task.  That  is,  the  alarm  does  not  directly 
impede  information  processing,  but  rather  vies  with  the  primary  task  for  the  operator’s  attention. 

An  example  of  this  might  be  a  flashing  border  on  all  four  sides  of  a  visual  display.  A  third  method 
of  interference  may  result  from  population  stereotypes  that  have  developed  concerning  what  alarms 
look  or  sound  like,  for  example  red  flashing  lights,  sirens  or  certain  iconic  symbols  (Z*a).  In  this 
case  it  is  assumed  that  the  semantic  content  associated  with  the  stimulus  immediately  overrides 
ongoing  processing  and  captures  attention.  In  other  words,  upon  perceiving  the  alarm,  the  operator 
immediately  knows  it  is  something  important  and  needs  attention. 

While  it  is  easy  to  see  how  such  models  can  suggest  the  intrusiveness  of  an  alert  in  such  obvious 
cases,  it  is  less  clear  how  they  would  predict  the  intrusiveness  of,  for  example,  a  ticker  style  text 
message  that  runs  along  the  bottom  of  the  screen. 
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The  general  guidance  that  can  be  taken  from  these  models  is  that  non-intrusive  alarms  should  not 
compete  for  sensory  or  cognitive  processing  capacity  of  the  primary  task,  should  not  divert 
attentional  resources  and  should  avoid  configurations  that  may  trigger  over-learned  orientation 
responses  because  of  familiarity  and  common  usage  in  society. 

In  developing  non-intrusive  interfaces  for  an  anomaly  alerting  system,  we  have  used  our 
knowledge  of  the  RJOC  operator’s  tasks  and  current  system  interface  to  inform  the  designs  such 
that  they  do  not  compete  for  attentional  resources  or  cognitive  processing  capacity  of  the  primary 
task.  However,  ultimately  the  degree  to  which  an  alert  provides  the  right  level  of  non-intrusiveness 
should  be  judged  and  validated  by  the  end  user  in  the  appropriate  operational  context. 

3.1 .2  Predictions  of  annoyance  models 

The  only  predictive  annoyance  models  that  we  have  found  deal  largely  with  noise  annoyance  and 
apply  to  more  general  societal  issues  than  to  cognitive  work  environments.  Such  models  may  take 
into  account  factors  of  the  strength  of  the  noise  source,  frequency,  adaptation  and  appraisal  by  the 
human,  emotions  aroused  and  cognitive  processing.  However,  we  have  not  found  any  extensive  or 
compelling  data  that  would  suggest  specific  design  guidance,  other  than  the  obvious.  For  example, 
repetitive  sounds  or  signals  will  generate  an  annoyance  factor  over  time.  However,  the  degree  to 
which  such  annoyance  produces  degradation  in  or  distraction  from  primary  task  performance 
cannot  be  specified,  nor  can  one  say  for  how  long  a  repetitive  signal  would  have  to  continue  before 
it  becomes  annoying,  what  level  of  intensity  causes  annoyance  and  how  individual  differences  and 
experience  may  influence  annoyance  “thresholds”. 

Therefore,  we  will  proceed  to  use  common  sense  in  the  interpretation  of  what  could  be  potentially 
annoying  in  an  alert  and  to  avoid  such  design  options.  However,  as  noted  above,  the  degree  to 
which  an  alert  provides  the  right  level  of  non-intrusiveness  should  ultimately  be  judged  and 
validated  by  the  end  user  in  the  appropriate  operational  context.  A  design  that  we  believe  to  be 
non-intrusive  could  ultimately  turn  out  to  be  the  opposite  if  the  underlying  anomaly  trigger  occurs 
with  high  frequency  in  actual  operations.  This  might  be  the  case  when  there  is  a  data  dump  from  an 
information  source  (e.g.,  PAL  flight  or  AIS  Vessel  Monitoring  System  (A VMS)  receiver). 

3.1 .3  Summary  of  principles 

The  following  are  the  main  principles  that  make  an  alert  intrusive. 

•  Direct  perceptual  interference  with  the  ongoing  primary  task; 

•  Direct  cognitive  interference  with  the  primary  task; 

•  Continued  presence  of  a  stimulus  that  by  repetitiveness,  intensity  or  quality  results  in  a 
psychological  or  emotional  state  that  causes  an  individual  to  lose  attentional  focus  on  the 
primary  task;  and 

•  The  greater  the  perceptual  deviation  of  the  alerting  stimulus  from  the  ambient  perceptual 
environment,  the  more  intrusive  it  will  generally  be  (e.g.,  providing  an  auditory  alert 
during  a  primarily  visual  task). 

It  follows  from  this  that  non-intrusive  alerts  must  have  properties  that  correspond  to  the  converse  of 
the  above  principles.  That  is,  a  non-intrusive  alert  must  not  directly  interfere  with  perceptual  or 
cognitive  processing  of  the  primary  task;  draw  the  operator’s  attention  away  from  the  primary  task 
due  to  its  repetitiveness,  intensity  or  quality;  or  differ  significantly  from  the  background  perceptual 
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environment.  However,  as  outlined  in  Section  5,  our  objective  was  to  design  non-intrusive  alerts 
for  anomalies  that  vary  in  priority  level;  from  low  (priority  3)  to  high  (priority  1).  Consequently, 
the  priority  of  the  anomalous  information  must  be  implied  by  the  level  of  intrusiveness  of  the  alert 
itself  That  is,  alerts  for  high  priority  anomalies  must  be  more  intrusive  than  those  for  low  priority 
information. 

In  addition  to  avoiding  the  above  factors  that  would  make  an  alert  intrusive,  a  key  element  in  alert 
system  design,  to  minimize  (if  not  avoiding  entirely)  interference  in  the  primary  task  due  to  alarm 
intrusiveness,  is  to  provide  an  ability  for  operators  to  match  alarm  priorities  to  the  appropriate  level 
of  cognitive  and  perceptual  intrusiveness. 
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4.  Literature  Review 


4.1  Method 

This  section  outlines  the  methodology  used  to  search  the  literature  as  well  as  select  research 
relevant  to  the  design  of  non-intrusive  alert  systems. 

4.1.1  Keywords 

The  following  keywords  for  conducting  the  literature  search  were  agreed  upon  with  the  Technical 
Authority  (TA)  and  are  shown  in  Table  1. 


Table  1:  Keywords 


Core  Concept 

Priority  1  keywords 

Priority  2  keywords 

Alerting  System  Technology 

Alarm/waming/alert  system 
Non-intrusive 

Advisory  system 

Tote  board 

Auditory/visual  alarm 

Tote 

Operator  Interface 

Design  guidelines/concepts 
Human-computer 

Operator-machine 

Interface  design 

User  interface 

Intelligen* 

Adaptive 

Etiquette 

Human  factors 

Smart 

Human  Performance 

False  alarm 

Attenti* 

Monitor* 

Workload 

Model* 

Situation*  awareness 

Distract* 

Disrupt* 

Anomaly/change  detection 

Anomal* 

Keywords  to  be  searched 
independently 

Alarm/waming/alert  overload 

Missed  alarm/warning/alert 

In  conducting  the  search,  all  Alerting  System  Technology  keywords  were  paired  with  Operator 
Interface  keywords  and  with  Human  Performance  keywords  using  “AND”  logic.  The  keywords  to 
be  searched  independently  (i.e.,  alarm/waming/alert  overload  and  missed  alarm/warning/alert) 
were  not  paired  with  any  other  keywords.  It  was  initially  proposed  that  if  the  literature  search  using 
the  priority  1  keywords  did  not  produce  ideal  results,  priority  2  keywords  would  be  substituted  for 
priority  1  keywords.  However,  based  upon  an  abundance  of  articles  found  with  priority  1  keywords 
there  was  no  reason  to  pursue  this  option. 
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4.1.2  Databases 


Table  2:  Databases  searched 


Database 

Description 

PsycINFO 

The  PsycINFO  database  is  a  collection  of  electronically  stored  bibliographic  references,  often  with 
abstracts  or  summaries,  to  psychological  literature  from  the  1800s  to  the  present.  The  available  literature 
includes  material  published  in  50  countries,  but  is  all  presented  in  English.  Books  and  chapters  published 
worldwide  are  also  covered  in  the  database,  as  well  as  technical  reports  and  dissertations  from  the  last 
several  decades. 

NTIS 

NTIS  is  an  agency  for  the  U.S.  Department  of  Commerce’s  Technology  Administration.  It  is  the  official 
source  for  government  sponsored  U.S.  and  worldwide  scientific,  technical,  engineering  and  business 
related  information.  The  400,000  article  database  can  be  searched  for  free  at  the  www.ntis.qov.  Articles 
can  be  purchased  from  NTIS  at  costs  depending  on  the  length  of  the  article. 

CISTI 

CISTI  stands  for  the  Canada  Institute  for  Scientific  and  Technical  Information.  It  is  the  library  for  the 

National  Research  Council  of  Canada  and  a  world  source  for  information  in  science,  technology, 
engineering  and  medicine.  The  database  is  searchable  on-line  at  cat.cisti.nrc.ca.  Articles  can  be 
ordered  from  CISTI  for  a  fee  of  approximately  $12. 

STINET 

STINET  is  a  publicly-available  database  (stinet.dtic.mil)  which  provides  access  to  citations  of  documents 
such  as:  unclassified  unlimited  documents  that  have  been  entered  into  the  Defence  Technology  Technical 
Reports  Collection  (e.g.,  dissertations  from  the  Naval  Postgraduate  School),  the  Air  University  Library 

Index  to  Military  Periodicals,  Staff  College  Automated  Military  Periodical  Index,  Department  of  Defense 

Index  to  Specifications  and  Standards,  and  Research  and  Development  Descriptive  Summaries.  The  full- 
text  electronic  versions  of  many  of  these  articles  are  also  available  from  this  database. 

Google 

Scholar 

The  World  Wide  Web  was  searched  using  the  Google  Scholar  search  engines  (scholar.google.com). 

DRDC 

Research 

Reports 

DRDC  Defence  Research  Reports  is  a  database  of  scientific  and  technical  research  produced  over  the 
past  6-  years  by  and  for  the  Defence  Research  &  Development  Canada.  It  is  available  online  at 
pubs.drdc-rddc.gc.ca/pubdocs/pcow1_e.html. 

4.1 .3  Search  strategy 

To  maintain  a  record  of  the  process,  the  following  information  was  documented  in  a  spreadsheet: 

•  Database  searched  (e.g.,  Psych  Info); 

•  Keyword  combination  (e.g.,  Non  intrusive  AND  attend*); 

•  Number  of  hits; 

•  Number  of  applicable  hits; 

•  Articles  downloaded; 

•  Articles/books  that  require  purchase;  and 

•  If  applicable,  where  in  the  article  the  keywords  were  searched  (i.e.,  only  in  the  article 
keywords  or  anywhere  in  the  article). 

4.1 .4  Selection  and  review  of  articles 

Two  articles  were  purchased  and  68  articles  were  downloaded  giving  a  total  of  70  relevant  articles. 
Each  article  was  then  briefly  reviewed  and  classified  according  to  article  theme  (i.e.,  Human 
performance  or  Operator  interface),  source  of  the  article  (i.e.,  keyword  search,  article  provided  by 
the  TA,  or  from  another  article),  and  whether  the  article  included  theoretical  or  empirical  research. 
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The  research  team  first  used  criteria  of  relevance  and  quality  to  evaluate  the  70  articles.  Relevance 
was  defined  as  how  closely  the  article  relates  to  the  research  objectives  outlined  in  the  Statement  of 
Work.  Specifically,  relevance  was  assigned  the  following  4  point  scale: 

•  1  =  non-intrusive  alerts  to  inform  operators  (design/performance)  mentioned  in  the  abstract 
or  paper; 

•  2  =  alerts  mentioned  in  abstract  or  paper; 

•  3  =  alerts  mentioned  in  abstract  or  paper,  but  not  the  focus;  and 

•  4  =  no  alerts  mentioned  in  abstract  but  still  somewhat  relevant. 

Quality  was  defined  in  terms  of  where  the  paper  was  published  using  a  3  point  scale  as  follows: 

•  1  =  journal; 

•  2  =  technical  report,  summary  or  conference  paper;  and 

•  3  =  magazine  article. 

Following  the  initial  search,  a  more  refined  search  was  undertaken  to  make  certain  all  relevant 
literature  was  obtained.  This  included  re-examining  all  the  articles  that  were  rated  4  V  in  terms  of 
relevance  (21  in  total)  and  identifying  any  relevant  references  and  additional  relevant  keywords 
that  were  not  included  in  our  original  keyword  list.  This  resulted  in  the  addition  of  the  following 
keywords: 

•  Notification  system; 

•  Alarm  cleanup; 

•  Nuisance  alarm;  and 

•  Pre-alarm. 

These  keywords  were  then  paired  with  all  Operator  Interface  and  Human  Performance  keywords 
and  searched  in  all  of  the  above  mentioned  databases. 

Finally,  the  research  team  conducted  a  search  for  any  articles  that  cited  the  21  most  relevant 
articles.  Based  on  this  refined  search,  4  new  articles  were  identified,  providing  a  total  of  74 
articles.  These  4  additional  articles  were  also  classified  and  evaluated  according  to  the  method 
described  above. 

An  EndNote  database  and  a  spreadsheet  with  a  record  of  all  74  articles,  classification  (as  described 
above),  relevance  and  quality  rating  as  well  as  any  relevant  notes  have  been  provided  to  the 
Scientific  Authority.  The  accompanying  endnote  database  includes  the  title,  year,  author,  journal 
and  abstract  for  the  74  articles. 

4 . 1.4. 1  Final  evaluation  criteria 

The  research  team  developed  the  final  criteria  to  evaluate  the  articles,  based  on  feedback  from  the 
TA.  The  main  concern  was  that  the  preliminary  criteria  would  not  capture  innovative  research 
from  other  areas  (other  than  the  core  alert/alarm  literature)  that  might  be  applicable  to  the  design 
of  non-intrusive  alerts.  Although  this  innovative  research  may  not  specifically  deal  with  alerts,  it 
could  be  applied  to  the  warning  system.  The  Quality  criteria  were  maintained  for  the  final  search 
but  the  relevance  criteria  were  refined  to  the  following  3  point  scale: 

•  1  =  concepts  related  to  alert  intrusiveness  (including  design/performance/models/theory) 
specifically  mentioned  in  paper; 

•  2  =  concepts  related  to  alerts  were  the  focus  of  paper;  and 

•  3  =  concepts  related  to  alerts  were  not  focus  of  paper. 
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Articles  were  then  re-evaluated  based  on  the  final  criteria.  As  a  result,  four  articles  that  originally 
had  a  lower  ranking  were  found  to  have  concepts  in  the  paper  related  to  alert  intrusiveness,  while  8 
articles  received  lower  relevance  scores. 

The  research  team  also  performed  an  additional  keyword  search  to  address  another  potentially 
relevant  area,  namely  human  centred  adaptive-automation.  Thus,  the  additional  following 
keywords  were  searched: 

•  Mission; 

•  Context; 

•  Operator;  and 

•  Adaptive. 

They  were  paired  with: 

•  Advisory  system; 

•  False  alarm;  and 

•  Intrusiveness. 

These  keywords  were  searched  in  Google  Scholar  resulting  in  two  new  articles  related  to  alert 
intrusiveness. 

In  total,  there  were  24  usable  articles  obtained  from  our  searches  that  are  related  to  alert 
intrusiveness.  These  articles  and  7  more  from  the  TA  made  up  the  31  articles  used  for  the  literature 
review. 

4.2  Results  of  the  Literature  Review 

In  this  section  we  provide  a  description  and  summary  evaluation  of  each  of  the  key  papers.  The 
papers  are  organized  first  by  functional  elements  of  an  alerting  system  (described  in  the  previous 
section).  Within  each  section,  the  literature  was  further  broken  down  into  the  four  main  categories 
described  previously  (i.e.,  conceptual  models,  generic  alert  system  design  guidelines,  specific 
design  concepts  for  alerts,  relevant  empirical  research  on  alert  systems). 

4.2.1  Configure  alert  parameters 

This  function  encompasses  the  creation  of  rules  for  determining  which  anomalous  events  will  give 
rise  to  an  alert,  and  assigning  an  alert  priority  for  each  rule  created. 

Of  the  four  categories  of  literature,  we  were  only  able  to  find  relevant  papers  relating  to  generic 
design  guidelines.  Note  that  some  additional  information  on  the  setting  of  trip  points  and  the 
reduction  in  nuisance  alerts  can  be  found  in  Brown,  O’Hara  and  Higgins  (2000)  which  is  reviewed 
in  detail  in  section  4. 2. 2. 2. 

4.2.1 .1  Design  Guidelines 
Configure  for  individual  differences 

As  computer  systems  requiring  adaptive  user  interface  technologies  mature  and  proliferate  into 
critical  applications,  Hudlicka  and  McNeese  (2002)  argue  that  it  is  critical  that  user  interfaces 
accommodate  individual  user  characteristics.  Citing  recent  research,  Hudlicka  and  McNeese  argue 
that  individual  differences  impact  perceptual  processes,  cognitive  processes,  motor  processes,  low- 
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level  processes  (e.g.,  attention  and  memory)  and  higher-level  processes  (e.g.,  situation  assessment, 
decision  making,  and  judgment).  In  fact,  they  state  that  ignoring  individual  differences  in  human- 
machine  systems  can  lead  to  non-optimal  behaviour  at  best  and  critical  errors  with  disastrous 
consequences  at  worst. 

With  respect  to  configuring  alert  parameters,  one  way  to  address  the  issue  of  individual  differences 
is  to  have  operators  enter  their  alert  preferences  directly  into  an  adaptive  interface  system. 
Hucklicka  and  McNeese  (2002)  developed  just  such  a  system  called  the  Affect  and  Belief  Adaptive 
Interface  System  (ABAIS).  ABAIS  allows  users  to  enter  display  and  modality  preferences  in  a 
number  of  categories,  which  can  be  seen  in  Table  3. 


Table  3:  Operator  information  preference  profile  (Hudlicka  &  McNeese,  2002) 


Information  Category 

Options 

Preferred  means  of  enhancing  visibility 

Colour,  Size,  Blinking 

Preferred  colour  for  alerts 

Red  solid,  Red  outline 

Preferred  alert  notification  modality 

Visual,  Auditory 

Preferred  attention  capture  means 

Movement  at  visual  periphery,  Shift  display  to 
foveal  region,  Enhance  icon  visibility,  Display 
arrow  pointing  to  desired  icon,  Superimpose 
blinking  icon 

Although  no  empirical  studies  have  been  conducted  using  ABAIS,  it  does  highlight  the  usefulness 
of  allowing  operators  to  control  the  way  in  which  alerts  are  configured.  Alert  settings  could  be 
based  on  operator  abilities.  For  example,  operators  who  have  auditory  difficulties  could  choose  to 
have  alerts  provided  visually,  or  operators  who  are  red/green  colour  blind  could  choose  alternate 
colour  schemes  to  report  the  importance  of  an  alert.  Alert  setting  could  also  be  based  on  operator 
preferences.  For  example,  it  could  be  that  one  operator  finds  a  blinking  cursor  annoying  and  would 
much  rather  have  an  audio  alert. 

Assessment 

While  this  paper  recognises  the  need  for  the  operator  to  configure  appropriate  alert  parameters,  it  is 
not  clear  that  allowing  individual  preferences  for  the  design  of  an  alert  would  be  feasible  in  team  or 
multi-operator  environments.  If  individual  members  of  the  RJOC  team  were  to  have  their  own  set 
of  preferences,  then  team  situation  awareness  for  the  current  alerts  status  would  be  compromised. 
Further,  supervisors  would  have  greater  difficulty  in  understanding  the  current  alert  picture.  There 
is  also  a  danger  that  individuals  left  to  choose  their  own  preferences  would  select  options  that  may 
be  inappropriate  from  a  Human  Factors  (HF)  perspective. 

Configure  for  context 

Providing  users  with  the  ability  to  configure  alert  parameters  can  also  be  important  in  the 
battlefield.  Commanders  and  staff  monitor  and  analyze  a  large  amount  of  digital  information  in  the 
midst  of  battle  planning,  preparation,  and  execution  in  order  to  understand  what  is  relevant  and 
critical  to  their  mission.  In  order  to  assist  the  decision-maker  to  understand  and  act  in  battlefield 
situations,  Akin,  Green  and  Amtz  (2005)  worked  to  develop  the  “System  to  Help  Identify  and 
Empower  Leader  Decisions”  (SHIELD).  SHIELD  monitors  digital  data  streams  and  alerts  a 
decision-maker  to  two  specific  battle  situations  requiring  immediate  action.  The  first  situation  is 
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when  a  friendly  unit  violates  a  boundary,  thus  risking  being  mistaken  for  enemy  elements  by 
friendly  units  (“Cross  Boundary  Violation  Alert”).  The  second  situation  is  when  a  unit’s  plan  for 
using  artillery  does  not  match  enemy  locations  determined  by  an  intelligence  section  (“Fratricide 
Prevention  Alert”).  In  both  situations,  SHIELD  initiates  an  alert  that  displays  a  text  warning  and  a 
flashing  icon  on  an  “alert  map”. 

Although  helpful,  Akin,  Green  and  Arntz  (2005)  note  that  alerts  can  be  intrusive.  They,  therefore, 
designed  SHIELD  to  allow  leaders  to  be  able  to  control  the  intrusiveness  of  alerts.  Leaders  are  able 
to  dismiss  an  alert  from  the  screen,  have  an  alert  be  repeated  at  what  may  prove  to  be  a  more 
convenient  time,  or  turn  off  an  alert.  In  addition  to  these  options,  they  plan  on  including  an 
intrusiveness  filter  in  future  versions  of  SHIELD.  This  intrusiveness  filter  would  allow  a  leader  to 
define  when  alerts  should  and  should  not  be  displayed  (e.g.,  within  a  certain  limit  of  known  threat 
locations;  if  less  than  3  alerts  are  already  displayed  on  the  screen).  The  intrusiveness  filter  would 
also  allow  leaders  to  turn  off  audio  alerts,  which  could  be  especially  important  if  there  is  a 
possibility  that  noise  could  signal  the  leader’s  presence  to  the  enemy. 

Assessment 

The  contribution  of  the  paper  with  respect  to  alert  configuration  is  useful  and  provides  a  good 
example  of  an  interface  that  allows  users  to  describe  contextual  conditions  for  when  an  alert  may 
be  presented.  It  should  be  noted,  however,  that  this  does  not  mitigate  the  actual  intrusiveness  of  that 
alert  when  it  does  occur. 

4.2.2  Receive  information  on  alert  state 

An  operator  receives  information  on  an  alert’s  state  through  an  alert  warning,  which  is  defined  as 
the  actual  mechanisms  by  which  an  operator’s  attention  is  captured.  Not  surprisingly,  literature 
relating  to  alert  warnings  was  by  far  the  most  abundant  of  any  of  the  alert  system  functions  and 
included  research  relating  to  HF  conceptual  models,  design  guidelines,  experiments,  and  design 
concepts. 

4.2.2. 1  HF  models 

McCrickard,  Chewar,  Somervell  andNdiwalana  (2003a)  suggest  that  alerting  systems  should 
deliver  current  and  critical  information  to  the  user  in  a  manner  where  the  user  is  not  interrupted 
from  their  primary  task.  It  is  assumed  that  alerting  systems  are  distracting  to  the  user,  however,  the 
authors  note  that  it  is  not  known  how  much  they  distract  the  user  and  if  this  distraction  is  negative. 
McCrickard  et  al.  (2003a)  examine  the  benefits  and  drawbacks  of  alert  system  interface  designs. 
Three  critical  parameters  for  modelling  alert  system  user  goals  and  system  designs  are  interruption , 
reaction  and  comprehension. 

Interruption  is  “an  event  promoting  transition  and  reallocation  of  attention  focus  from  a  task  to  the 
notification”  (p.  319).  Context  plays  an  important  part  in  interruption.  For  instance,  if  the  user  is 
driving,  the  alert  (i.e.,  check  engine  light)  should  not  be  intrusive  and  interruptive  to  the  main  task 
at  hand  (i.e.,  driving  safely).  However,  in  the  context  of  the  RJOC,  an  alert  informing  an  operator 
of  a  vessel  behaving  anomalously  may  require  immediate  attention  and  interruption  of  the  main 
task  at  hand. 

Reaction  “is  the  rapid  and  accurate  response  to  the  stimuli  provided  by  notification  systems”  (p. 
319).  Reaction  can  be  measured  by  the  time  it  takes  to  shift  attention,  which  can  be  manipulated 
through  designs  incorporating  colours,  shapes  and  motion. 
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Comprehension  is  “remembering  and  making  sense  of  the  information”  (p.  319)  which  can  be 
measured  through  recall  and  content  related  questions. 

Using  this  model,  combinations  of  high  (1)  and  low  (0)  interruption,  reaction  and  comprehension 
(IRC)  were  created  to  categorize  the  types  of  alerts,  as  shown  in  Figure  2. 


Reaction 


Comprehension 


Figure  2:  IRC  framework  (adapted  from  McCrickard  et  al.,  2003a,  p.  321) 

The  following  table  is  a  list  of  potential  alerts  and  examples  from  alert  systems  according  to  the 
IRC  framework. 
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Table  4:  IRC  framework  alerts  and  examples  (McCrickard  et  al.,  2003a) 


Types  of  Alerts 

IRC  Codes 

Examples 

Critical  activity 
monitor 

1H3 

A  system  administrator  uses  a  network  monitor  on  the  desktop 
to  enable  prompt  responses  and  fixes  to  computer  problems 

Alarm 

110 

A  businessman  relies  on  a  calendar  and  email  alerts 

Information  exhibit 

101 

A  factory  supervisor  requires  critical  updates,  while  performing 
daily  tasks 

Secondary  display 

Oil 

An  editor  writes  part  of  a  document  while  monitoring  a  team 
progress  groupware  tool 

Diversion 

100 

A  teenager  on  the  home  computer  enjoys  amusing  pop-ups 

Indicator 

010 

A  tourist  uses  a  GPS  (Global  Positioning  System)  to  navigate 
around  a  new  city 

Ambient  media 

001 

A  telemarketer  without  a  window  has  changing  weather 
desktop  wallpaper 

Noise 

000 

A  student  doing  homework  is  reassured  with  internet  radio 

This  table  displays  the  alerts  in  terms  of  most  interruptive,  comprehensive,  and  requiring  reaction, 
to  the  least  interruptive,  comprehensive,  and  requiring  reaction.  Using  Wickens  and  Hollands 
(2000)  human  information  processing  stage  model,  McCrickard  et  al.  (2003a)  mapped  each  of  the 
IRC  categories  onto  the  information  processing  model.  This  enabled  the  authors  to  illustrate  that 
different  alert  types  have  different  information  processing  routes.  For  instance,  when  people 
process  noise  (low  interruption,  reaction  and  comprehension),  it  does  not  reach  their  working 
memory  as  compared  to  a  diversion  (high  interruption,  low  reaction  and  comprehension)  which 
does  reach  working  memory.  The  two  most  relevant  alert  types  that  produce  a  low  interruption,  but 
require  some  type  of  comprehension,  are  ambient  media  and  a  secondary  display.  Ambient  media 
is  processed  through  the  senses  and  then  enters  long  term  working  memory.  However,  in  the  case 
of  a  secondary  display,  a  selection  response  must  be  executed  in  order  to  get  feedback.  This  means 
that  a  secondary  display  requires  more  effort  on  the  part  of  the  operator,  but  not  as  much  as  a  high 
interruption,  reaction  and  comprehension  alert  (i.e.,  critical  activity  monitor)  would  require. 

McCrickard  et  al.  (2003a)  performed  a  case  study  to  test  the  IRC  category  framework.  The  case 
study  involved  two  separate  evaluations,  using  questionnaire  measures,  to  determine  effectiveness. 
The  first  study  was  conducted  at  Microsoft  with  the  Scope  alert  system  (van  Dantzich,  Robbins, 
Horvitz  &  Czerwinski,  2002;  as  cited  in  McCrickard  et  al.,  2003a).  The  second  study  was 
conducted  in  McCrickard  et  al.’s  lab  using  the  IRC  framework.  Although  the  Scope  was  meant  to 
be  used  differently  than  a  standard  interface,  the  questionnaire  for  the  Microsoft  study  had  items 
assessing  standard  interface  issues.  Therefore,  the  results  of  the  Microsoft  study  were  not  useful  in 
identifying  design  strategies.  The  questionnaire  for  the  lab  study,  however,  was  based  on  the  IRC 
framework.  That  is,  the  questions  assessed  tradeoffs  between  interruption,  reaction  and 


3  For  example,  a  critical  activity  monitor  would  be  high  in  interruption  (II),  high  in  reaction  (Rl)  and  high  in 
comprehension  (Cl),  while  an  alarm  would  be  high  in  intermption  (II),  high  in  reaction  (Rl)  and  low  in 
comprehension  (CO). 
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comprehension.  The  questionnaire  based  on  the  IRC  framework  produced  better  redesign 
strategies. 

The  Scope  is  an  alerting  system  which  is  located  in  the  bottom  right  comer  of  a  computer  screen 
and  presents  symbols  according  to  urgency  (urgent  items  are  located  closer  to  the  middle  than  non¬ 
urgent  items).  For  an  illustration  of  the  Scope  alerting  system,  see  McCrickard  et  al.  (2003a) 

The  Scope  is  divided  into  four  sections:  email  inbox,  calendar,  task  list  and  general  alerts.  The 
goals  of  the  Scope  system  are  to  direct  user  attention  to  urgencies  and  minimize  user  attention  for 
incoming  non-urgent  alerts.  According  to  the  IRC  framework,  the  Scope  would  be  similar  to  an 
alarm  (IRC  110)  and  ambient  display  (IRC  001).  The  results  of  the  studies  and  ensuing  guidelines 
are  summarized  in  McCrickard  et  al.  (2003a). 

The  full  explanation  of  the  Scope  can  be  found  in  van  Dantzich  et  al.  (2002;  as  cited  in  McCrickard 
et  al.  2003a).  McCrickard  et  al.  (2003a)  conclude  that  the  IRC  framework  provides  a  method  to 
evaluate  alert  systems.  The  Scope  is  one  of  these  alert  systems  that,  based  on  the  evaluation, 
yielded  potential  redesign  strategies  such  as  reducing  visual  clutter  and  defining  alerts  based  on 
shape  and  colour. 

Assessment 

In  summary,  the  McCrickard  et  al.  (2003a)  model  is  based  on  interruption  of  the  alert,  reaction  to 
the  alert  and  comprehension  of  the  alert.  These  parameters  create  8  different  types  of  alerts  which 
vary  on  the  IRC  levels.  For  an  alert  to  be  non-intrusive  it  would  need  to  be  non-interruptive.  A 
secondary  display  and  ambient  media  are  two  types  of  alerts  that  may  offer  a  less  intrusive  way  to 
inform  the  operator  of  interesting  events.  Ambient  media  allows  for  comprehension  of  the  material 
presented,  without  reaction.  Ambient  media  would  be  useful  in  the  context  of  non-urgent  alerts  or 
alerts  that  do  not  require  an  action.  A  secondary  display  for  alerts  may  not  be  a  viable  design 
solution  for  RJOC  as  presently  configured. 

In  terms  of  applicability  to  maritime  anomaly  alerts,  the  Scope  approach  may  be  too  extensive  for 
the  existing  RJOC  design  space,  may  require  too  much  operator  cognitive  processing  and  may 
provide  more  information  than  is  required. 

Overall,  this  paper  provides  a  good  conceptual  analysis  of  the  human  information  processing  issues 
that  need  to  be  considered  with  respect  to  the  intrusiveness  and  comprehensibility  of  warning 
indicators. 

4. 2.2.2  Design  guidelines 

Nuisance  alerts 

In  the  1980s,  reports  of  air  and  train  crashes  came  to  light  indicating  that  system  operators  turned 
off  critical  alert  or  warning  indicators  prior  to  the  accidents.  In  his  seminal  paper,  Sorkin  (1988) 
outlined  two  reasons  why  he  believes  operators  would  disable  warning  signals.  The  first  reason  is 
that  alert  signals  can  be  highly-aversive  and  can  interfere  with  important  operator  duties  (e.g.,  high- 
level  shrill  sound  produced  by  warning  whistles,  multiple  alerts  that  make  it  difficult  to  identify  the 
underlying  condition  for  the  alerts).  The  second  reason  is  that  operators  perceive  false  alarm  rates 
to  be  excessively  high.  A  common  sense  analysis  indicates  that  operators  with  high  workloads  will 
adopt  strategies  to  ignore  or  disarm  excessive  alerts,  especially  if  they  are  perceived  to  have  high 
false  alarm  rates.  To  deal  with  these  issues,  Sorkin  recommended  a  number  of  changes  to  alert 
systems,  which  include: 
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•  Design  alert  signals  in  such  a  way  that  they  are  not  overly  aversive  or  disruptive. 
Possible  alert  techniques  suggested  include  speech  message  alerts  and  special  alert 
codes. 

•  System  designer  should  consider  the  effect  of  the  alert  rate  on  the  performance  of  the 
entire  alert  system,  especially  when  the  operator  has  a  heavy  workload. 

Assessment 

Sorkin’s  (1988)  reasons  for  operators  turning  off  the  alerts  are  still  applicable  twenty  years  later. 
Constant  false  alarms  provide  a  false  sense  of  security  to  an  operator  whereby  it  is  assumed  the 
alerts  are  not  important  enough;  thus,  they  do  not  receive  the  operator’s  attention.  Also,  designing  a 
less  intrusive  alerting  method  can  prevent  distraction  from  the  primary  task.  There  was 
considerable  research  in  the  area  of  design  guidelines  relating  to  alert  warnings;  it  will  be  discussed 
in  the  following  sections. 

General  alert  principles 

Shorrock,  Scaife  and  Cousins  (2002)  developed  high-level  principles  for  the  design  of  soft-desk 
alert  systems  that  could  contribute  to  a  philosophy  of  alert  handling.  The  principles  were  derived 
from  information  gained  from  two  studies  regarding  the  design  and  evaluation  of  Air  Traffic 
Management  (ATM)  commercial-off-the-shelf  (COTS)  systems  software  and  hardware.  These 
studies  resulted  in  approximately  100  recommendations  for  alert  handling  system  design.  The 
recommendations  were  generalized  into  a  smaller  set  of  design  principles,  which  were  structured 
according  to  Stanton’s  (1994;  as  cited  in  Shorrock,  Scaife  &  Cousins,  2002)  model  of  alert-initiated 
activities. 

Stanton’s  model  consists  of  six  activities: 

1 .  Observation:  the  detection  of  an  abnormal  condition  within  the  system.  At  this  stage,  care 
should  be  taken  to  ensure  that  colour  and  flash/blink  coding  support  alert  monitoring  and 
searching.  Excessive  use  of  colour  and  blinking  can  de-sensitize  an  operator  and  reduce  the 
ability  of  the  alert  to  gain  the  operator’s  attention. 

2.  Acceptance:  acknowledging  receipt  and  awareness  of  an  alert.  Alert  systems  should  reduce 
operator  workload  to  a  manageable  level  -  excessive  demands  for  acknowledgement 
increase  workload  and  operator  error. 

3.  Analysis:  prioritization  of  an  alert  based  on  task  and  system  in  which  it  occurs. 

4.  Investigation:  activities  that  aim  to  discover  the  underlying  cause  of  an  alert  in  order  to 
deal  with  the  fault. 

5.  Correction:  the  application  of  the  results  from  the  previous  stages  to  address  the  problem 
identified  by  an  alert. 

6.  Monitoring:  assessment  of  the  outcome  from  the  Correction  stage. 

Shorrock  et  al.  (2002)  added  co-ordination  as  a  seventh  activity  to  the  model.  Co-ordination  is  the 
transfer  of  information  between  operators  and  the  application  of  collaborative  efforts  to  observe, 
accept,  analyze,  investigate  or  correct  faults.  Their  final  list  of  principles  for  human-centred  alert 
systems  classified  according  to  the  revised  Stanton  model  can  be  seen  in  Table  5. 
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Table  5:  Model-based  principles  for  human-centred  alarm  systems 

(Shorrock  et  al.,  2002) 


Observe 

Alarms  should  be  presented  (time  stamped)  in  chronological  order,  and  recorded  in  a  log  in  the  same  order. 

Alarms  should  signal  the  need  for  action. 

Alarms  should  be  relevant  and  worthy  of  attention  in  all  the  plant  states  and  operating  conditions. 

Alarms  should  be  detected  rapidly  in  all  operating  conditions. 

It  should  be  possible  to  distinguish  alarms  immediately,  i.e.  different  alarms,  different  operators,  alarm  priority. 

The  rate  at  which  alarm  lists  are  populated  must  not  exceed  the  users’  information  processing  capabilities. 

Auditory  alarms  should  contain  enough  information  for  observation  and  initial  analysis  and  no  more. 

Alarms  should  not  annoy,  startle  or  distract  unnecessarily. 

An  indication  of  the  alarm  should  remain  until  the  operator  is  aware  of  the  condition. 

The  user  should  have  control  over  automatically  updated  information  so  that  information  important  to  them  at  any 
specific  time  does  not  disappear  from  view. 

It  should  be  possible  to  switch  off  an  auditory  alarm  independent  of  acceptance,  but  it  should  repeat  after  a 
reasonable  period  if  the  fault  is  not  fixed. 

Failure  of  an  element  of  the  alarm  system  should  be  made  obvious  to  the  operator. 

Accept 

Reduce  the  number  of  alarms  that  require  acceptance  as  far  as  is  practicable. 

Allow  multiple  selection  of  alarm  entries  in  alarm  lists. 

It  should  be  possible  to  view  the  first  unaccepted  alarm  with  a  minimum  of  action. 

In  multi-user  systems,  only  one  user  should  be  able  to  accept  and/or  clear  alarms  displayed  at  multiple 
workstations. 

It  should  only  be  possible  to  accept  the  alarm  from  where  the  sufficient  alarm  information  is  available. 

It  should  be  possible  to  accept  alarms  with  a  minimum  of  action  (e.g.  double  click),  from  the  alarm  list  or  mimic. 

Alarm  acceptance  should  be  reflected  by  a  change  on  the  visual  display,  such  as  a  visual  marker  and  the 
cancellation  of  attention-getting  mechanisms,  which  prevails  until  the  system  state  changes. 

Analyse 

Alarm  presentation,  including  conspicuity,  should  reflect  alarm  priority,  with  respect  to  the  severity  of 
consequences  associated  with  the  delay  in  recognising  the  deviant  condition. 

When  the  number  of  alarms  is  large,  provide  a  means  to  filter  the  alarm  list  display  by  sub-system  or  by  priority. 

Operators  should  be  able  to  suppress  or  shelve  certain  alarms  according  to  system  mode  and  state,  and  see 
which  alarms  have  been  suppressed  or  shelves,  with  facilities  to  document  the  reason  for  suppression  or  shelving. 

It  should  not  be  possible  for  operators  to  change  priorities  of  any  alarms. 

Automatic  signal  over-riding  should  always  ensure  that  the  highest  priority  signal  over-rides. 

The  coding  strategy  should  be  the  same  or  all  display  elements. 

Facilities  should  be  provided  to  allow  operators  to  recall  the  position  of  a  particular  alarm  (e.g.  periodic  divider 
lines). 

Alarm  information  such  as  terms,  abbreviations  and  message  structure  should  be  familiar  to  operators  and 
consistent  when  applied  to  alarm  lists,  mimics  and  message/event  logs. 

The  number  of  coding  techniques  should  be  at  the  required  minimum,  but  dual  (redundant)  coding  may  be 
necessary  to  indicate  alarm  status  and  improve  accurate  analysis  (e.g.  symbols  and  colours). 

Alarm  information  should  be  positioned  so  as  to  be  easily  read  from  the  normal  operating  position. 

Investigate 

Alarm  point  information  (e.g.,  settings,  equipment  reference)  should  be  available  with  a  minimum  of  action. 

Information  on  the  likely  cause  of  an  alarm  should  be  available. 

A  detailed  graphical  display  pertaining  to  a  displayed  alarm  should  be  available  with  a  single  action. 

When  multiple  display  elements  are  used,  no  individual  element  should  be  completely  obscured  by  another. 

Visual  mimics  should  be  spatially  and  logically  arranged  to  reflect  functional  or  naturally  occurring  relationships. 

Navigation  between  screens  should  be  quick  and  easy,  requiring  a  minimum  of  user  action. 

Correct 
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Every  alarm  should  have  a  defined  response  and  provide  guidance  or  indication  of  what  response  is  required. 

If  two  alarms  for  the  same  system  have  the  same  response,  then  consideration  should  be  given  to  grouping  them. 

It  should  be  possible  to  view  status  information  during  fault  correction. 

Use  cautions  for  operations  that  might  have  detrimental  effects. 

Alarm  clearance  should  be  indicated  on  the  visual  display,  both  for  accepted  and  unaccepted  alarms. 

Local  controls  should  be  positioned  within  reach  of  the  normal  operating  position. 

Monitor 

No  primary  principles.  However,  a  number  of  principles  primarily  associated  with  observation  become  relevant  to 
monitoring. 

Coordinate 

Provide  high-level  overview  displays  to  show  location  of  operators  in  system,  areas  of  responsibility,  etc. 

4. 2. 2. 2. 1.1  Assessment 

These  studies  provide  a  solid  and  useful  analysis  of  the  major  principles  that  should  be  considered 
in  the  design  of  alert  systems,  many  of  which  are  applicable  to  considerations  for  non-intrusive 
systems.  Although  not  the  focus  of  these  papers,  no  specific  implementation  guidelines  are 
provided  for  how  these  principles  should  be  addressed  in  terms  of  concrete  design  concepts  or 
considerations. 

Auditory  alerts 

Auditory  warnings  are  used  as  a  means  of  commanding  attention  without  startling  or  annoying 
operators.  They  are  used  in  domains  such  as  aviation,  medicine,  and  control  rooms.  Such  warnings 
are  meant  to  convey  information  about  a  task  or  situation,  and  often  must  compete  with  other 
sensory  stimuli  in  the  environment.  However,  there  are  problems  associated  with  exposure  to  loud 
noise  over  prolonged  periods,  including  hearing  loss,  cardiovascular  problems  (Babisch,  1998;  as 
cited  in  Edworthy  &  Hellier,  2002),  cognitive  problems  and  performance  problems  (World  Health 
Organisation,  1993;  Smith,  1993;  Edworthy,  1997;  as  cited  in  Edworthy  &  Hellier,  2002).  In  their 
review  of  research  on  auditory  warnings  in  noisy  environments,  Edworthy  and  Hellier  (2002)  make 
the  following  suggestions  regarding  how  to  deal  with  issues  related  to  auditory  alerts  (see  Table  6). 

Ahlstrom  (2003)  also  conducted  work  to  develop  guidelines  for  auditory  alerts  within  the  National 
Airspace  System  (NAS).  New  tools  for  the  NAS  are  accompanied  by  new  equipment  using  visual 
and  auditory  signals  to  indicate  the  status  of  equipment  and  of  incoming  air  traffic.  The  auditory 
signals  are  often  developed  with  minimal  use  of  standards  or  guidelines,  or  with  minimal 
consideration  of  the  auditory  signals  on  existing  equipment.  This  can  result  in  too  many  warnings, 
warnings  that  are  too  loud  or  inaudible,  warnings  that  are  confusing,  and  inappropriate  mapping 
between  the  warning  and  its  meaning  (Meredith  &  Edworthy,  1994;  as  cited  in  Ahlstrom,  2003). 
Such  situations  can  result  in  operators  disarming  or  inhibiting  alerts,  as  described  by  Sorkin  (1988). 

Ahlstrom  (2003)  conducted  an  exploratory  study  to  provide  increased  understanding  and  insight 
into  audio  alert  problems  within  the  NAS.  After  conducting  a  literature  review,  Alhstrom  identified 
15  issues  associated  with  auditory  alerts  in  a  variety  of  fields  (e.g.,  hospital  emergency  rooms, 
aircraft  cockpits).  Twenty  current  and  former  terminal  Air  Traffic  Control  Specialists  (ATCS)  were 
then  asked  to  rate  each  of  the  15  issues  on  an  1 1  point  scale  ranging  from  0  (not  a  problem)  to  10 
(severe  problem)  on  how  problematic  the  issue  was  from  them  in  their  work  environment. 
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Table  6:  Suggestions  for  auditory  alerts  (Edworthy  &  Hellier,  2002) 


Issues 

Suggestions 

Noisy  environments 
(e.g.,  factories, 
cockpits,  flight  decks) 

As  ambient  noise  determines  the  detectability  of  auditory  warnings,  a  worst-case  spectrum 
should  be  used  to  avoid  excessively  loud  warnings 

Ensure  warnings  are  not  too  loud  as  they  may  be  switched  off  by  operators 

Avoid  having  a  number  of  warnings  going  off  at  the  same  time  as  they  will  mask  each  other 

Acoustic 

Localizability  will  be  improved  by  having  several  audible  components  in  the  warning  sound, 
preferably  with  a  fairly  low  fundamental  frequency 

Continuous  tones  should  not  be  used  as  warning  sounds 

Speech  vs.  non¬ 
speech 

It  is  more  difficult  to  produce  intelligible  speech  warnings  for  a  complex  noise  environment  than 
it  is  to  fit  a  non-speech  warning  to  the  same  noise  spectrum 

Speech  warnings  can  create  problems  in  certain  environments.  For  example,  a  verbal  warning 
in  a  hospital  ward  may  cause  worry  or  panic  by  patients  and  relatives,  whereas  a  nonverbal 
warning  would  not 

Warning  sounds  can  be  designed  to  mimic  speech  in  some  way.  For  example,  in  an  emergency 
room  environment  the  warning  sound  for  “cardiovascular”  can  contain  6  pulses,  the  same  as  the 
word,  with  a  pitch  pattern  consisting  of  3  pulses  at  one  pitch  followed  by  3  at  a  lower  level.  Such 
a  warning  is  easily  learned  and  retained 

Speech  warnings  may  interfere  with  ongoing  information  processing  which  involves  language 
(Wickens  &  Hollands  Information  Processing  Model,  2000) 

Warning  design 
protocols 

Warnings  must  be  designed  so  as  to  attract  attention  without  causing  stress  and  annoyance 

False  alarms 

Design  alert  systems  with  high  accuracy  rates.  False  alarms  provide  information  of  no  use 
whatsoever,  serve  to  increase  the  noise  levels  within  an  environment,  and  over  time  will  lower 
performance  on  a  task 

Number  of  auditory 
warnings 

Individually  different  alerts  for  top-priority  situations  only 

Use  specific  sounds  to  identify  the  category  of  risk 

Focus  on  functions  rather  than  equipment.  For  example,  assign  specific  alerts  to  specific 
medical  functions  rather  than  to  pieces  of  equipment  (as  equipment  changes  frequently  in 
health  care) 

Participants  were  also  provided  the  following  comments  on  specific  problems  with  auditory  alerts 
in  their  work  environment: 

•  Alerts  can  be  annoying  at  times  (e.g.,  alert  continues  to  sound  even  after  it  has  been 
located); 

•  Alerts  for  different  problems  sound  similar  and  are  easily  confused; 

•  Too  many  false  alarms.  After  a  while,  alerts  start  to  lose  their  meaning; 

•  Alerts  can  interfere  with  voice  communications; 

•  Some  systems  should  have  auditory  alerts  but  do  not; 

•  Alerts  that  are  too  loud  cause  stress,  frustration  and  hearing  loss. 
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In  response  to  these  findings,  Ahlstrom  (2003)  provided  recommendations  to  be  considered  for 
future  alerts  used  by  terminal  area  operators: 

•  Alerting  and  warning  systems  should  be  unambiguous,  with  a  clear  indication  of  the  cause 
for  the  alert.  This  can  be  accomplished  by  using  varying  frequency  and/or  modulation,  and 
periodicity  should  differ  (i.e.,  avoid  continuous  signals); 

•  The  criteria  for  conditions  causing  frequent  false  alarms  should  be  evaluated  and  effort 
should  be  made  to  reduce  the  number  of  irrelevant  alerts; 

•  Systems  should  have  a  simple,  consistent  means  for  turning  off  non-critical  auditory  alerts 
without  erasing  any  displayed  message  accompanying  the  auditory  signal.  The  system 
could  also  include  a  sensing  mechanism  that  automatically  shuts  off  the  auditory  portion  of 
an  alert  when  it  no  longer  provides  useful  information; 

•  Strategies  should  be  used  to  maximize  the  ability  to  locate  auditory  alerts.  These  include 
avoiding  mid  frequencies,  positioning  alerts  to  the  side  rather  than  in  the  front  or  in  the 
back  of  the  operator,  minimizing  hard  surfaces  in  order  to  decrease  echoes,  and  providing  a 
centralized  alert  panel  or  window  indicating  the  alert  status  for  most  alerts; 

•  When  operators  are  required  to  identify  an  alert  based  on  the  auditory  portion  alone,  the 
number  of  signals  to  be  identified  should  not  exceed  four.  Up  to  nine  alerts  can  be  used  if 
they  are  presented  regularly,  and  up  to  12  alerts  can  be  used  if  relative  discrimination 
(rather  than  absolute  identification)  is  used. 

4. 2. 2. 2. 1.2  Assessment 

These  papers  outline  many  of  the  critical  design  and  implementation  issues  with  respect  to  auditory 
alerts.  The  first  three  of  Ahlstrom’s  recommendations  for  auditory  alerts  would  also  be  generically 
applicable  to  the  implementation  of  other  forms  of  alerting. 

Nuclear  control  room 

Within  the  nuclear  industry,  the  goal  of  computer-based  alert  systems  is  to  assist  operators  by 
processing  alert  data  and  improving  the  presentation  of  alert  information.  Brown,  O’Hara  and 
Higgins  (2000)  conducted  an  investigation  to  update  and  revise  the  Nuclear  Regulatory 
Commission’s  (NRC)  guidance  for  reviewing  alert  system  designs.  The  resulting  revisions  were 
based  on  NRC  research  on  the  effects  of  alert  system  design  characteristics  on  operator 
performance  and  on  research  examining  the  introduction  of  computer-based  human-system 
interface  systems  into  Nuclear  Power  Plants.  Some  of  the  revised  guidelines  that  would  be 
applicable  to  naval  anomaly  detection  include  setpoint  determination,  assured  functionality,  alert 
reduction,  alert  signal  validation,  parameter  stability,  and  alert-status  separation. 

Setpoint  Determination  and  Nuisance  Alert  Avoidance 

The  determination  of  alert  setpoints  should  consider  the  trade-off  between  the  timely  alerting  of  an 
operator  to  off-normal  conditions  and  the  creation  of  nuisance  alerts  caused  by  establishing 
setpoints  so  close  to  the  “normal”  operating  values  that  occasional  excursions  of  no  real 
consequence  are  to  be  expected. 
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Assured  Functionality  under  High  Alert  Condition 

The  alert  processing  system  should  ensure  that  alerts  which  require  immediate  operator  action  or 
indicate  a  threat  to  safety  functions  are  presented  in  a  manner  that  supports  rapid  detection  and 
understanding  by  the  operator  under  all  alert  loading  conditions. 

Alert  Reduction 

The  number  of  alert  messages  presented  to  the  crew  during  off-normal  conditions  should  be 
reduced  by  alert  processing  techniques  (from  a  no-processing  baseline)  to  support  the  crew’s  ability 
to  detect,  understand,  and  act  upon  alerts  that  are  important  to  the  plant  condition  within  the 
necessary  time. 

Alert  Signal  Validation 

Sensor  and  other  input  signals  should  be  validated  to  ensure  that  spurious  alerts  are  not  presented  to 
plant  personnel,  due  to  sensor  or  processing  system  failure. 

Parameter  Stability  Processing 

The  alert  system  should  incorporate  the  capability  to  apply  time  filtering,  time  delay,  or 
deadbanding4  to  the  alert  inputs  to  allow  filtering  of  noise  signals  and  to  eliminate  unneeded 
momentary  alerts. 

Alert-Status  Separation 

Status  indications,  messages  that  indicate  the  status  of  plant  systems  but  are  not  intended  to  alert 
the  operator  to  the  need  to  take  action,  generally  should  not  be  presented  via  the  alert  system 
display  because  they  increase  the  demands  on  the  operators  for  reading  and  evaluating  alert  system 
messages. 

4. 2. 2. 2. 1.3  Assessment 

Although  the  nuclear  industry  is  different  from  the  operational  environment  of  the  RJOC,  many  of 
the  same  issues  may  be  applicable,  for  example,  Brown  et  al.’s  (2000)  recommendations  relating  to 
reducing  the  number  of  alerts,  rapid  detection  of  alerts,  and  presenting  only  meaningful  alerts. 

These  recommendations  would,  in  theory,  reduce  the  workload  of  the  operator  and  make  the 
overall  alerting  system  less  intrusive. 

Graphic  user  interface  design  for  alerts 

A  man-machine  interface  (MMI)  is  the  way  in  which  an  operator  controls  a  system  and 
traditionally  contains  manual  buttons,  controls,  switches,  and  a  monitor  through  a  closed-circuit 
television.  When  switching  to  a  graphic  user  interface  (GUI)  from  a  MMI,  some  mistakes  can 
occur  that  can  adversely  impact  human  performance.  A  common  mistake  is  the  MMI  display  is 
shrunk  onto  the  GUI.  Although  it  maintains  familiarity  for  the  user,  it  can  result  in  usability  issues, 
which  can  lead  to  efficiency  and  safety  problems. 

Han,  Yang  and  Im  (2007)  created  a  six-phase  approach  to  develop  a  method  for  GUI  design.  The 
six  phases  include: 

1 .  UI  design  guidelines  collection  for  process  control  rooms5 


4  Deadband  is  a  specified  area  where  the  alert  would  not  go  off. 
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2.  Usability  inspection5 

3.  Design  rules  and  guidelines  development 

4.  User  interface  design  and  prototyping 

5.  Usability  testing  and  evaluation 

6.  Final  prototypes  and  design  specs. 

For  the  first  step,  Han  et  al.  (2007)  conducted  an  extensive  review  of  design  guidelines  for  any 
GUIs  used  in  a  process  control  room.  Consequently,  they  organized  approximately  1500  guidelines 
into  14  chapters  and  70  sections.  These  guideline  principles  covered  topics  such  as  aesthetics, 
attention,  cognitive  issues,  consistency,  display  issues,  feedback,  forgiveness,  memory  issues, 
metaphors,  simplicity,  system  messages  and  help,  user  control,  and  user  differences. 

Han  et  al.  (2007)  then  analyzed  current  user  interfaces  in  a  process  control  room  and  operator  tasks 
to  identify  usability  problems  and  define  design  requirements  for  a  new  interface.  This  new  process 
control  room  interface  prototype  takes  into  account  all  the  current  problems  experienced  by 
operators.  Operator  manuals  were  reviewed  and  operators  were  interviewed.  Twenty-two  operators 
participated  in  one-on-one  interviews  answering  questions  assessing  tasks,  alerts  systems  and 
requirements.  Overall,  the  usability  inspection  resulted  in  the  identification  of  500  usability  problems 
by  4  practitioners.  The  most  frequently  found  problems  related  to  attention  (e.g.,  warning  signals  not 
salient),  consistency  (e.g.,  different  terminologies),  cognitive  issues  (e.g.,  pictures  on  screens 
different  from  actual  layout  of  equipment),  memory  issues  (e.g.,  same  colours  have  different 
meanings)  and  simplicity  (e.g.,  too  many  colours  used).  Selected  problems  are  shown  in  Table  7. 


Table  7:  Identified  problems  and  design  requirements  (Han  et  al.,  2007) 


Alert  System  Problems 

Usability  Problems 

Operator’s  Requirements 

Warnings  are  not  categorized  by 
urgency 

Screens  are  complex 

Unnecessary  or  redundant  interface 
elements  should  be  removed,  and  the 
layout  should  be  reorganized 

Irregular  situations  may  not  be 
informed  to  operators 

Each  screen  has  different  layouts  and 
different  methods  of  information 
presentation  or  visualization 

Consistent  screen  layouts  and 
visualization  should  be  designed 

Warning  signals  are  not  attracting  the 
operator’s  attention 

Too  many  colors  are  used  on  a  screen 

The  color-coding  scheme  should  be 
developed,  and  the  meaning  of  colour 
should  be  easily  understandable 

Visual  alerts  are  not  presented  with 
auditory  alerts 

Only  one  window  can  be  activated  at  a 
time 

The  multi-window  function  should  be 
available 

Only  the  mouse  can  be  used  to  move 
between  input  fields 

A  ‘Tab’  key  should  also  be  available 
as  a  method  of  moving  between  input 
fields 

System  status  changes  cannot  be 

detected  before  opening 

and  looking  at  related  screens  directly 

All  the  status  changes  should  be 
automatically  informed  on  the  screen 
that  the  operators  mainly  use 

5  This  step  can  be  skipped  if  a  requirements  analysis  already  exists 
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Once  the  usability  problems  and  operator  requirements  were  identified,  Han  et  al.  (2007)  created 
design  rules  and  specific  design  guidelines  for  the  new  interface.  Design  rules  are  defined  as  major 
premises  that  designers  should  always  keep  in  mind  when  designing.  Guidelines  refer  to  the 
specific  methods  designers  should  follow  when  designing  individual  interface  elements.  Design 
rules  were  categorized  into  improvement  of  task  efficiency  or  reduction  of  task  errors.  Each  rule 
was  accompanied  with  several  specific  design  guidelines.  Design  guidelines  generated  by  Han  et 
al.  (2007)  addressed  critical  problems  of  alert  design  and  colour-coding,  as  shown  in  Table  8. 


Table  8:  Examples  of  guidelines  (Han  et  al.,  2007) 


Design  Categories 

Guidelines 

Alert  Design 

Attention 

Warning  should  be  easily  distinguished  from  the  background.  For  example,  visual 
signals  should  be  bright  on  a  dark  screen,  and  vice  versa 

Hazard  information 

A  warning  signal  should  contain  hazard  information.  However,  not  too  much  hazard 
information  should  be  included  in  just  one  signal 

Consequences  information 

Consequences  information  should  follow  the  hazard  information.  However,  if  operators 
can  infer  the  consequences  from  the  hazard  information,  they  do  not  need  to  be 
included  in  the  warning  signal 

Instructions 

Information  on  instructions  should  not  include  a  too  difficult  or  impossible  method  to 
perform.  That  is,  instruction  methods  as  easy  as  possible  should  be  included 

Comprehension 

If  operators'  capability,  knowledge,  and  experience  levels  are  various,  warning  signals 
should  be  easy  so  that  operators  who  have  the  lowest  capability  or  experience  level  can 
understand  them 

Motivation 

Warning  signals  should  induce  operators  to  read  or  listen  to  them  and  to  react  to  them 

Brevity 

Alert  information  should  not  exceed  two  phrases  or  sentences 

Durability 

Warnings  that  inform  operators  of  the  instantaneous  change  of  process  or  signals  that 
do  not  need  special  reactions  should  not  last  for  a  long  time 

Colour  Design 

General  considerations 

Colours  are  used  for  supporting  search  tasks,  highlighting,  or  indicating  status  of  the 
system 

Foreground  colour 

Do  not  use  blue,  magenta,  or  a  shade  of  pink  in  displaying  information  that  the  user 
must  read 

Colour  contrast 

Exaggerate  lightness  differences  between  foreground  and  background  colours,  and 
avoid  using  colours  of  similar  lightness  adjacent  to  one  another,  even  if  they  differ  in 
saturation  or  hue 

Colours  of  interface  elements 

Do  not  design  a  colour  icon  that  is  substantially  different  from  the  black-and-white  icon. 
When  a  colour  is  added  to  an  icon,  it  is  best  to  leave  the  one-pixel  black  outline  and 
other  black  lines  that  form  the  icon,  and  fill  the  icon  in  with  colour 

As  can  be  seen  in  Table  8,  Han  et  al.  (2007)  provide  a  number  of  recommendations  for  alert  system 
design.  For  example,  warnings  should  be  contrasted  with  background  information  (e.g.,  bright 
warning  on  a  dark  screen,  vice  versa),  and  should  be  colour  coded  (red,  yellow,  green,  white),  but 
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not  in  shades  of  blue  or  pink.  Important  for  reducing  the  intrusiveness  of  alerts  are  the 
recommendations  that  information  in  the  warnings  should  be  easily  interpretable  and  should  not 
exceed  2  sentences,  and  warnings  that  do  not  require  an  operator’s  reaction  should  not  last  a  long 
time. 

4. 2. 2. 2. 1.4  Assessment 

Although  the  recommendations  and  guidelines  outlined  by  Han  et  al.  (2007)  are  not  specific  to 
creating  less  intrusive  alerts,  the  following  recommendations  can  be  generalized  for  the  design  of 
less  intrusive  alerting  systems: 

•  Decrease  the  disruptiveness  and  intrusiveness  of  alerts. 

•  Consider  the  effect  of  the  alert  rate  on  the  operator.  Alert  processing  techniques  should  be 
used  to  reduce  the  number  of  alert  messages  as  this  will  improve  ability  to  detect, 
understand  and  act  upon  important  alerts. 

•  Be  cautious  of  alert  set  points.  They  should  be  set  at  a  level  to  avoid  nuisance  and  false 
alarms.  False  alarms  do  not  provide  useful  information  and  over  time  will  lower 
performance  on  a  task. 

•  Do  not  present  status  indications  through  an  alert  system  display.  This  increases  the 
demands  on  an  operator  for  reading  and  evaluating  the  message. 

•  Auditory  alerts  should  use  specific  sounds  to  identify  the  category  of  risk.  This  will  aid  an 
operator’s  ability  to  identify  alerts. 

•  Operators  should  be  able  to  turn  off  non-critical  alerts  without  erasing  information. 

•  The  auditory  portion  of  an  alert  should  shut-off  automatically  when  it  no  longer  provides 
useful  information. 

•  Operators  should  be  able  to  distinguish  alerts  immediately  (e.g.,  different  alerts,  alert 
priority). 

•  The  rate  at  which  alert  lists  are  populated  must  not  exceed  the  users’  information 
processing  capabilities. 

•  Alert  acceptance  should  be  reflected  by  a  change  on  the  visual  display,  such  as  a  visual 
marker  and  the  cancellation  of  attention-getting  mechanisms,  which  prevails  until  the 
system  state  changes. 

•  Alert  presentation,  including  conspicuity,  should  reflect  alert  priority,  with  respect  to  the 
severity  of  consequences  associated  with  delay  in  recognizing  the  deviant  condition. 

•  Operators  should  be  able  to  suppress  or  shelve  certain  alerts  according  to  system  mode  and 
state,  and  see  which  alerts  have  been  suppressed  or  shelved,  with  facilities  to  document  the 
reason  for  suppression  or  shelving. 

•  A  detailed  graphical  display  pertaining  to  a  displayed  alert  should  be  available  with  a 
single  action. 

It  should  be  noted  that  many  of  the  recommendations  are  in  the  form  of  general  principles  (e.g., 
matching  alert  conspicuity  with  alert  priority)  and  there  is  a  lack  of  specific  recommendations  on 
how  the  principle  would  be  implemented  in  an  interface. 
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4 . 2. 2. 3  Empirical  research 

This  section,  which  reviews  empirical  research  related  to  alert  warnings,  is  divided  into  visual, 
visual  and  auditory,  and  predictive  alerting  system,  which  relates  to  the  actual  alerting  system  itself 
and  how  it  can  predict  the  operator’s  actions. 

Visual  Alerts 

There  are  many  variations  of  visual  alerts  such  as  text,  picture,  symbols  and  icons  which  may  vary 
in  size,  colour  and  position.  While  developing  a  pre-alert  system  that  would  reduce  the  frequency 
of  alerts,  Hwang,  Lin,  Liang,  Yenn  and  Hsu  (2008)  examined  whether  these  pre-alerts  should  be 
text  or  graphic.  The  idea  behind  testing  both  formats  was  to  determine  if  an  operator  is  more 
inclined  to  disregard  one  format  compared  to  another  (e.g.,  too  annoying,  not  comprehensive,  not 
noticeable),  which  would  in  turn  lead  to  more  alerts  going  off.  The  results  of  this  experiment  (see 
Section  4. 2. 4. 2  Warning  of  impending  alert)  showed  no  significant  differences  in  the  text  and 
graphic  pre-alert  types  for  reducing  the  number  of  alerts.  However,  with  the  graphic  types,  the 
operators  had  significantly  more  correct  answers  when  asked  questions  about  the  alerted  task. 
Although  both  forms  of  pre-alert  systems  would  be  a  benefit  to  the  operator,  the  graphic  display 
includes  more  information,  but  requires  more  space  and  greater  changes  to  be  implemented  in  the 
control  room.  Therefore,  Hwang  et  al.  (2008)  recommended  the  text  type  pre-alert  to  be 
implemented  in  control  rooms. 

McCrickard,  Catrambone,  Chewar  and  Stasko  (2003b)  considered  variations  of  animated  text  for 
computer  alert  systems.  Specifically,  McCrickard  et  al.  (2003b)  examined  the  visual  aspect  of  an 
alert,  and  its  effect  on  interruption,  reaction,  and  comprehension6.  There  were  three  forms  of 
animated  texts:  a  smooth  ticker  (information  shifted  horizontally),  a  fading  display  (information 
fades),  and  a  rapid  serial  visual  presentation  (RSVP)-style  “blast”  (displays  information  without 
smooth  animation).  Information  came  in  the  form  of  changing  news,  weather,  stocks  and  sports 
information.  The  primary  task  for  participants  was  searching  the  World  Wide  Web  for  information 
to  answer  questions  that  were  asked  of  them.  The  alerted  task  was  to  monitor  the 
news/weather/stocks/sports  information  (presented  by  animated  text)  and  answer  questions  relating 
to  the  message  displayed.  For  example,  while  participants  were  trying  to  answer  the  question,  “In 
what  year  was  Mount  Rushmore  carved,”  they  also  had  to  monitor  the  weather  alerts  and  press  a 
button  when  the  weather  temperature  dropped  below  30  degrees.  At  the  end  of  the  session, 
participants  were  asked  to  complete  awareness  questions  which  assessed  the  amount  of  information 
they  recalled  from  the  alerted  task  (i.e.,  monitoring  news/weather/stocks/sports  information). 
Results  did  not  report  a  specific  text  type  which  yielded  the  fastest  response  times,  rather  strengths 
and  weaknesses  were  found  for  each  method  of  animation  (see  Table  9). 


6  Interruption,  reaction  and  comprehension  were  defined  previously  in  Section  4.2.1. 
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Table  9:  Results  for  operator  tasks:  Study  1  (McCrickard  et  al.,  2003b) 


Tasks 

Measurement 

Best 

Worst 

Browsing  speed 

The  time  the  information  appeared  on  the  screen 
until  the  participant  typed  in  the  correct  answer 
and  pressed  OK 

Control  then  ticker 

Fade  &  blast 

Browsing 

comprehension 

The  number  of  incorrect  answers 

Control  then  blast 

Ticker  &  fade 

Link  selections 

The  number  of  times  a  participant  pressed  the 

Back  button 

Ticker  then  fade 

Control  &  blast 

Reaction  time  to  alert 

The  time  the  alert  appeared  on  the  screen  until 
the  participant  acknowledged  it  by  pressing  a 
button 

Blast  (34  seconds) 

Ticker  (54  seconds) 

Basic  awareness  hit  rate 

Recognition  of  information  in  the  alert 

Ticker 

Fade 

Detailed  awareness  rate 

The  recognition  of  correct  and  incorrect  answers 

Ticker 

Fade  &  blast 

Basic  awareness  false 
alarm  rate 

Information  participants  reported  seeing  that  was 
not  actually  presented 

Fade 

Ticker 

Detailed  awareness 
false  alarm  rates 

Confidence  in  understanding  information  that  was 
not  actually  understood 

Ticker 

Fade  &  blast 

Participant’s  preference 

Most  user  friendly  and  least  intrusive 

Ticker 

Blast 

McCrickard  et  al.  (2003b)  recommended  the  ticker  as  the  best  choice  for  maintaining  awareness, 
minimizing  interruption,  facilitating  reaction  and  facilitating  comprehension. 

A  second  experiment  by  McCrickard  et  al.  (2003b)  examined  the  impact  that  alert  display  size  and 
animation  speed  would  have  on  performance.  This  experiment  only  used  the  ticker  text  and  fade 
text,  as  the  blast  type  was  rated  as  the  least  favourite  by  participants  in  the  first  experiment.  Results 
are  shown  in  Table  10. 
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Table  10:  Results  for  operator  tasks:  Study  2  (McCrickard  et  al.,  2003b) 


Tasks 

Measurement 

Best 

Worst 

Browsing  speed 

The  time  the  information  appeared  on  the  screen 
until  the  participant  typed  in  the  correct  answer 
and  pressed  OK 

Slow  ticker  or  any  slow 

Small  ticker  or  any 
small 

Browsing 

comprehension 

The  number  of  incorrect  answers 

Small  fade 

Small  ticker 

Link  selections 

The  number  of  times  a  participant  pressed  the 

Back  button 

- 

Small  ticker 

Reaction  time  to  alert 

The  time  the  alert  appeared  on  the  screen  until 
the  participant  acknowledged  it  by  pressing  a 
button 

Fade 

Normal  or  slow  ticker 

Basic  awareness  hit  rate 

Recognition  of  information  categories  in  the  alert 

Small  ticker 

Slow  ticker 

Detailed  awareness  rate 

The  recognition  of  correct  and  incorrect  answers 

Small  fade 

Small  ticker  &  fade 

Basic  awareness  false 
alarm  rate 

Information  participant's  reported  seeing  that  was 
not  actually  presented 

Slow  ticker 

Small  ticker  &  fade 

Detailed  awareness 
false  alarm  rates 

Confidence  in  understanding  information  that  was 
not  actually  understood 

Slow  fade 

Small  fade 

Participant’s  preference 

Most  user  friendly  and  least  intrusive 

- 

- 

Based  on  these  results,  McCrickard  (2003b)  recommended  that  the  slow  fade  may  be  the  best 
overall  alert  type. 

In  summary,  all  three  text  types  did  not  significantly  interrupt  the  user  from  the  primary  task,  but 
still  alerted  them  to  important  information.  Fade,  blasts,  and  small  displays  were  better  for  rapid 
identification  of  the  information  compared  to  tickers  or  other  size  displays,  but  worse  for 
comprehension  and  recall.  Overall,  alerts  that  travel  across  the  screen  horizontally  were  found  to  be 
the  least  intrusive  and  yield  the  best  performance  results.  However,  when  the  alert  text  was  varied 
in  size  and  speed,  the  text  that  appeared  slowly  and  faded  slowly  was  the  best  type  to  use.  The 
finding  with  respect  to  ticker  style  alerts  provides  guidance  to  the  development  of  a  similar  design 
concept  for  the  present  project. 

The  operational  definition  of  alert  intrusiveness  in  terms  of  the  relationship  between  a  primary  task 
and  the  alerting  stimulus  is  useful  and  will  assist  in  the  development  of  non-intrusive  alerts 
concepts. 

Visual  and  Auditory 

The  literature  is  mixed  regarding  performance  for  salient  (e.g.,  auditory)  versus  less  salient  (e.g., 
visual)  cues.  Banbury,  Macken,  Tremblay  and  Jones  (2001;  as  cited  in  Colcombe  &  Wickens, 

2006)  found  that  discrete  auditory  stimuli,  such  as  those  used  in  alerts,  tend  to  corrupt  memory 
processes  more  than  visual  cues.  On  the  other  hand,  Helmick-Rich,  Burke,  Gilad  and  Hancock 
(2004;  as  cited  in  Colcombe  &  Wickens,  2006)  found  that  people  were  more  likely  to  comply  with 
an  auditory  cue  compared  to  a  visual  cue.  It  therefore  appears  that  the  degree  of  disruption  due  to 
alerted  tasks  is  a  complicated  relationship  between  the  characteristics  of  the  ongoing  task,  the 
alerted  task,  and  the  operator’s  strategies  and  skills.  Auditory  alerts  appear  to  be  more  interrupting 
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than  visual  alerts,  but  do  not  always  impose  a  cost  to  ongoing  tasks.  Colcombe  and  Wickens  (2006) 
were  interested  in  the  way  in  which  parameters  of  a  primary  task  made  them  more  or  less 
interruptible  from  an  alert,  and  in  turn,  how  the  characteristics  of  the  alert  mediated  the  costs  of 
interruptions  to  primary  tasks. 

For  study  1,  participants  were  warned,  visually  or  auditorily,  for  potential  collision  threats  from  a 
Cockpit  Displays  of  Traffic  Information  (CDTI)  display.  Participants  were  in  one  of  the  following 
conditions:  stable  tracking,  unstable  tracking  condition,  binary  alert  (normal  state  or  high-level 
alert),  likelihood  alert  (normal  state,  mid-level  or  high-level  alert).  Results  showed  participants 
were  faster  to  detect  auditory  alerts  than  visual  alerts.  Tracking  performance  was  better  in  the 
binary  alerting  condition  than  the  likelihood  alerting  condition.  Subsequent  studies  replicated  the 
binary  and  likelihood  finding,  but  found  response  times  faster  for  visual  than  auditory  alerts. 

Colcombe  and  Wickens  (2006)  stated  that  the  binary  alert  was  generally  more  effective  than  the 
likelihood  alert.  However,  this  was  based  on  participants  responding  to  the  binary  alert  faster  than 
the  mid-level  likelihood  alert,  without  taking  into  consideration  comprehension  or  interruption.  No 
urgency  distinction  was  made  with  binary  alerts  and,  thus,  operators  must  treat  each  alert  as  if  it  is 
high-level.  Auditory  alerts  generally  supported  better  conflict  detection  response  than  did  a  visual 
alert.  However,  the  impact  of  modality  on  the  concurrent  task  was  modulated  by  the  nature  of  the 
task  (visual  tracking  was  more  disrupted  by  the  auditory  alert  than  by  a  visual  alert). 

Similarly,  Krausman,  Elliot  and  Pettitt  (2005)  examined  visual,  auditory  and  tactile  alerts  on 
platoon  leader  decision  making.  To  a  limited  extent,  the  military  has  implemented  a  multi-sensory 
information  presentation  approach  in  that  system  designers  are  using  auditory  and  visual  displays. 
However,  there  are  situations  in  which  a  soldier’s  visual  and  auditory  channels  are  heavily  loaded. 
Therefore,  tactile  presentations  may  also  be  beneficial.  Krausman  et  al.  (2005)  had  twelve  infantry 
officers  participate  in  3  different  scenarios.  In  each  scenario  participants  played  the  role  of  platoon 
leaders  (PL)  mounted  inside  a  vehicle.  During  the  scenario,  participants  sat  in  front  of  a  primary 
display,  map  display,  and  Unmanned  Aerial  Vehicle  (UAV)  display  to  perform  tactical 
communications  and  monitor  activity  on  the  display.  Approximately  9  of  the  communications  sent 
to  the  participant  in  each  scenario  were  preceded  by  a  visual,  audio,  or  tactile  alert.  Table  1 1 
describes  the  presentation  of  each  alert. 


Table  1 1 :  Recommendations  for  design  for  a  platoon  leader  alert  (Krausman  et  al., 

2005) 


Alert  purpose 

Alert  presentation 

To  alert  platoon 
leader  to  an 
incoming 
message 

Visual  alert  -  solid  red  box  appears  on  bottom  portion  of  communications  console  of  primary 
display 

Auditory  alert  -  “beep”  from  headset 

Tactile  alert  -  “buzz”  from  tactical  armband 

Participants  received  only  one  type  of  alert  in  each  scenario  (e.g.,  visual  alerts  in  scenario  1,  audio 
alerts  in  scenario  2,  and  tactile  alerts  in  scenario  3).  Alerts  were  continuous  and  stopped  when  the 
participant  clicked  the  “show  message”  button  to  receive  the  information.  After  each  scenario, 
participants  rated  and  ranked  the  effectiveness,  helpfulness,  and  necessity  of  the  alerts. 

Overall,  visual  alerts  were  the  least  effective  method  of  alerting  participants.  Response  times  were 
significantly  longer  for  visual  alerts  than  for  auditory  or  tactile  alerts;  no  significant  differences 
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were  found  between  auditory  and  tactile  alert  response  times.  Participants  also  rated  visual  alerts  as 
the  less  effective  method  of  getting  their  attention.  When  ranking  alerts,  participants  ranked 
auditory  alerts  to  be  the  most  effective  method  of  getting  their  attention  followed  by  tactile  alerts 
and  visual  alerts,  respectively.  Tactile  alerts  were  ranked  as  the  most  helpful  type  of  alert,  followed 
by  auditory  alerts  and  visual  alerts,  respectively.  However,  participants  noted  that  caution  should 
be  exercised  when  implementing  auditory  and  tactile  alerts  in  combat  vehicles  because  alerts  might 
be  hard  to  detect  in  combat  environments  (e.g.,  multiple  radio  nets  might  mask  the  sound  of  an 
auditory  alert,  tactile  alerts  might  be  missed  in  a  moving  vehicle  due  to  vehicle  vibrations). 
Participants  suggested  that  a  combination  of  alerts  might  be  the  most  effective  option. 

Steefkerk,  Esch-Bussemakers  and  Neerincx  (2007)  designed  a  context-aware  alert  system.  This 
system  varied  visual  and  auditory  methods,  and  reported  that  participants  preferred  an  auditory 
signal  for  low  urgency  messages.  More  details  about  this  study  can  be  found  in  Section  4. 2. 3. 2. 

In  summary,  the  experiments  investigating  visual  alerts  recommend  alerts  that  have  a  slow  fade 
text.  Visual  alerts  that  are  not  recommended  include  the  blast  alert  and  graphical  pre-alerts.  In 
general,  it  was  found  that  auditory  alerts  support  better  detection  than  visual  alerts.  Experiments 
investigating  auditory  alerts  recommend  an  auditory  alert  for  all  types  of  urgencies,  and  that  these 
alerts  vary  the  presentation  based  on  urgency. 

Predictive  Alerting  System 

While  the  previous  sections  addressed  visual  and  auditory  alerts,  this  section  presents  empirical 
research  related  to  a  predictive  alerting  system,  which  can  aid  the  operator  in  certain  tasks.  Mitchell 
(1998)  considered  the  issues  surrounding  intelligent  aids  and  associates  in  an  operational  setting. 
Intelligent  aids  and  associates,  such  as  displays  of  up-to-date  information  about  operations,  are  a 
type  of  computer  technology  that  is  designed  to  help  operators.  Mitchell  states  that  an  operator’s 
immediate  response  after  hearing  an  alert  may  be  to  shut  it  off,  followed  by  identifying  what 
triggered  the  alert.  Often  times  software  updates  follow  well  behind  changes  in  operational 
requirements  such  that  by  the  time  software  updates  are  finally  introduced,  certain  alerts  may  have 
become  extinct  or  irrelevant.  A  common  issue  is  that  alerts  are  assumed  to  be  extinct  or  irrelevant, 
thus  are  ignored  by  operators.  To  address  these  issues,  Mitchell  (1998)  designed  a  system  to 
prevent  alert  overload,  in  turn,  increasing  alert  usefulness. 

The  Operator  Function  Model  Expert  System  (OFMspert)  performs  many  functions,  but  the  most 
relevant  to  the  present  project  is  the  activity  tracking  device,  whereby  the  system  tracks  the 
operator’s  activities.  The  computer  would  track  the  activity  by  asking  “ What  is  the  operator  doing? 
Why  is  the  operator  doing  that?  What  will  the  operator  do  next?”  (p.  31).  The  latter  question  was 
posed  as  a  way  for  the  system  to  predict  or  infer  the  intent  of  the  operator.  Actions  that  can  be 
interpreted  include  cognitive  actions  (i.e.,  situation  assessment)  and  perceptual  actions  (i.e., 
scanning  for  alerts).  Although  not  discussed  in  the  paper,  the  system  could  presumably  predict  the 
operator’s  future  actions,  thus  presenting  an  alert  at  an  appropriate  time.  An  alert  that  is  presented 
to  the  operator  at  an  appropriate  time,  would  not  only  be  less  intrusive,  but  also  more  useful. 
Another  useful  capability  of  the  system  is  that  it  can  explain  recommendations  to  an  operator, 
which  could  reduce  the  operator’s  cognitive  workload. 

The  National  Aeronautics  and  Space  Administration  (NASA)  Goddard  Space  Flight  Center 
(GSFC)  was  chosen  to  illustrate  the  application  of  the  OFMspert.  The  Multisatellite  Operations 
Control  Center  (MSOCC)  is  a  system  in  GSFC  that  monitors  the  use  and  effectiveness  of  computer 
systems  shared  by  satellites.  Several  steps  were  taken  for  the  design  methodology,  but  the 
implementation  of  intelligent  aids  and  associates  was  of  interest  for  this  review.  Specifically, 
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inferring  the  intent  of  the  operator  to  reasonably  predict  their  activities  and  interpret  their  actions 
was  examined.  The  OFMspert  implementation  in  MSOCC  was  studied  by  Jones,  Mitchell  and 
Rubin  (1990;  as  cited  in  Mitchell,  1998).  The  experiment  evaluated  the  previous  Saisi  and  Mitchell 
MSOCC  data  and  verbal  protocols  from  two  control  participants.  Every  action  had  an  interpretation 
from  the  OFMspert,  the  participants  or  a  domain  expert  (from  the  Saisi  and  Mitchell  MSOCC 
data).  Results  showed  significantly  good  matches  for  system  commands  and  display  requests. 
However,  there  were  poor  matches  for  planning  and  browsing.  Another  implementation  of  the 
OFMspert  was  performed  by  Callantine,  Mitchell  and  Palmer  (1997;  1998)  and  looked  at  the 
Boeing  B-757  flight  deck  (as  cited  in  Mitchell,  1998).  Results  showed  the  OFMspert  correctly 
interpreted  92%  of  the  actions,  when  compared  with  10  certified  pilots.  Those  that  were  not  correct 
related  to  browsing  actions. 

Assessment 

This  section,  which  describes  empirical  research  relating  to  alert  warnings,  considers  both  the 
visual  and  auditory  components  of  an  alert,  as  well  as  how  to  reduce  the  operator’s  workload.  For 
the  visual  component  of  the  alert,  a  slow  fade  text  is  recommended.  For  the  auditory  component  of 
the  alert,  a  sound  for  low,  medium  and  high  urgencies  should  be  presented  with  the  alert  and  the 
sound  should  vary  according  to  urgency.  Lastly,  there  is  a  recommendation  of  having  a  predictive 
adaptive  system  which  can  infer  the  actions  of  the  operator.  This  prediction  can  allow  the  operators 
to  be  interrupted  with  a  medium  or  low  urgency  alert  during  periods  of  low  workload. 

The  OFMspert  technology  is  a  decade  old  and  perhaps  could  be  modified  to  fit  the  needs  of  the 
maritime  domain.  Presumably,  if  this  system  can  track  the  activities  of  an  operator  and  predict  what 
actions  they  will  perform,  specifically  alert  scanning;  it  can  have  some  merit  as  part  of  an  alerting 
system.  Predicting  operator’s  actions  in  tandem  with  an  alerting  system  could  predict  when  the 
operator’s  workload  could  be  interrupted  with  an  alert.  This  is  especially  the  case  when  the 
operator  is  already  scanning  for  alerts.  Combining  the  OFMspert  with  a  warning  system  could  be 
beneficial  in  that  operators  are  being  alerted  when  their  workload  allows  for  it,  and  the  rest  of  their 
time  is  spent  on  the  primary  task,  except  during  cases  of  emergency.  However,  the  major  limitation 
of  this  approach  concerns  the  accuracy  with  which  the  system  is  able  to  determine  the  suitable  time 
for  presenting  alert  information.  Even  small  errors  would  likely  result  in  operators  becoming 
frustrated  with  low  level  alerts  that  arrive  when  they  are  busy  and  important  alerts  arriving  too  late. 

4.2.2A  Design  concepts 

The  literature  provided  a  number  of  specific  design  concepts  for  reducing  the  intrusiveness  of  alert 
warnings.  These  concepts  are  related  to  information  presentation,  limiting  operator  information 
overload,  maintaining  situation  awareness,  and  reducing  the  number  of  alerts. 

Information  presentation 

A  key  design  for  reducing  the  intrusiveness  of  alerts  can  be  found  in  Hautamaki,  Bagnall  and 
Small’s  (2006)  research  on  the  Hazard  Monitor  and  Intelligent  Alerting  System  (HMIAS).  This 
system  was  designed  to  improve  information  presentation  to  combat  system  (CS)  operators  using 
the  Combat  Control  System  (CCS).  The  CCS  was  designed  to  help  CS  operators  form  a  tactical 
picture  of  the  maritime  environment,  especially  surface  and  subsurface  vessel  locations.  Included 
in  the  CCS  is  an  alert  system  to  notify  operators  of  conditions  that  violate  expected  operating 
ranges.  Originally  the  alert  system  indicated  isolated  incidents  but  did  not  convey  the  severity  of 
the  situation  as  a  whole  to  the  operator.  Over  the  years,  improvements  to  the  CCS  have 
incorporated  a  great  deal  of  disjointed  but  related  information.  However,  Hautamaki  et  al.  (2006) 
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point  out  that  there  is  still  room  for  improvement.  In  particular,  the  researchers  argue  that  the  Alert 
Manager  window  on  the  Tactical  Control  and  Weapons  Control  interface  has  a  method  for 
presenting  alerts  that  is  too  subtle.  This  subtle  alerting  mechanism  has  led  to  situations  in  which 
operators  were  so  focused  on  an  ongoing  task  that  they  failed  to  notice  other  safety  alerts.  In 
addition,  Hautamaki  et  al.  (2006)  argue  that  the  CCS  method  of  organizing  safety  alerts  by 
occurrence  or  contact  number  makes  it  difficult  for  operators  to  identify  the  most  severe  alerts. 

To  address  these  issues,  Hautamaki  et  al.  (2006)  developed  the  Hazard  Monitor  and  Intelligent 
Alerting  System  (HMIAS)  to  improve  the  overall  CCS.  HMIAS  monitors  for,  prevents,  traps,  and 
captures  operator  errors  in  order  to  prevent  the  negative  consequences  associated  with  errors  (see 
Figure  3  for  a  generic  Hazard  Network). 


Generic  CCS  Operator  Error/Hazard  Network 


Situation: 

Non-urgent  external  conditions; 
Operator  at  low  workload,  stress 


Task  begun 


Situation: 

Increasing  urgency;  Operator 
at  moderate  workload,  stress 


Initiate  task 

Non-intrusive  alerts 

Cautions, 
some  aiding 

Warnings, 
more  aiding 

Consequences 

avoided 

k  A 

Situation: 

Urgent;  Operator  at 
high  workload,  stress 


Benign  Situations  - 


►Moderate  Situations - 


Increasing  severity  of  consequences 
Increasing  intrusiveness  of  alerts 


Figure  3:  Hazard  Network  (Hautamaki  et  al.,  2006,  p.  7) 

HMIAS  monitors  system  states  for  hazards,  and  alerts  operators  to  the  hazards  in  a  timely,  context 
sensitive  and  multi-modal  manner.  For  example,  an  initial  alert  is  presented  in  the  form  of  text  on 
the  operator’s  screen.  If  the  alert  is  not  acknowledged  in  a  sufficient  time  period,  and  the  condition 
persists  or  worsens,  the  alert  is  presented  as  flashing  text,  which  then  proceeds  to  an  audible  alert, 
followed  by  the  addition  of  verbal  instructions. 

To  study  the  effectiveness  of  HMIAS,  Hautamaki  et  al.  (2006)  simulated  a  mission  in  which  sonar 
personnel,  fire  control  personnel  (including  CS  operators  and  CS  supervisor),  and  the  Officer  of  the 
Deck  in  a  Virginia  Class  submarine  control  room  track  an  unfriendly  quiet  diesel  submarine 
through  a  strait  while  remaining  undetected.  During  the  mission,  operators  encounter  commercial 
vessels,  deep-draft  tankers,  and  fishing  trawlers.  After  20  minutes,  the  scenario  concludes  when  a 
controlled  close  aboard  encounter  with  a  deep-draft  tanker  requires  an  evasive  manoeuvre. 
Participants  were  tasked  with  continuously  hunting  for  the  best  system  solution  for  contacts  using 
target  motion  analysis.  Those  assigned  to  the  baseline  condition  used  only  current  CCS  alerts, 
whereas  those  assigned  to  the  experimental  condition  used  CCS  alerts  with  HMIAS  technology. 

Results  indicate  that  HMIAS  enhances  operator  performance.  What  is  of  particular  importance  to 
less  intrusive  alert  technology  is  the  HMIAS  hazard  network.  As  events  monitored  by  the  system 
increase  in  severity  of  consequences,  the  intrusiveness  of  the  alerts  also  increase.  This  allows  the 
operator  to  easily  assess  the  urgency  of  an  alert  and  be  able  to  quickly  and  easily  identify  the  alerts 
that  require  immediate  attention. 
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McFarlane  and  Latorella  (2002)  identified  some  less  intrusive  methods  for  presenting  alert 
warnings  in  their  review  of  interruption  management  literature.  Interruptions  are  prevalent  in  many 
working  environments  in  which  humans  and  computers  interact  with  reactions  being  both  positive 
and  negative.  For  example,  interruptions  can  provide  important  information,  but  they  can  also 
cause  stress  and  hinder  performance.  McFarlane  and  Latorella  (2002)  discuss  the  Aegis  weapon 
system  used  by  the  navy  as  an  example.  This  system  interrupts  users  through  an  alert  tool  that 
presents  messages  and  task  assignments  on  an  ongoing  basis.  Although  operators  must  be  informed 
of  the  alerts,  the  alerts  are  in  fact  interruptions,  occurring  several  per  minute  during  high-stress 
operations.  McFarlane  and  Latorella  (2002)  provide  guidelines  based  upon  Latorella’s  Interruption 
Management  Stage  Model  (1996,  1998;  as  cited  in  McFarlane  &  Latorella,  2002).  This  model 
explores  the  process  of  human  interruption  in  a  work  environment,  as  shown  in  Figure  4. 
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Figure  4:  Interruption  Management  Stage  Model  (adapted  from  McFarlane  & 

Latorella,  2002,  p.  16) 

McFarlane  and  Latorella  (2002)  propose  four  design  solutions  to  deal  with  interruption:  immediate 
interruption,  negotiated  interruption,  mediated  interruption,  and  scheduled  interruption. 

•  Immediate  interruption  is  required  by  some  tasks.  When  tasks  require  this  type  of  interruption, 
some  implementations  may  make  it  easier  for  the  operator  to  resume  his  or  her  primary  task. 
For  instance,  Lee  (1992;  as  cited  in  McFarlane  &  Latorella,  2002)  found  that  an  active  window 
with  an  animated  border  produced  less  confusion  upon  resuming  a  task  than  an  active  window 
with  a  fixed  border.  Similar  to  this,  Davies,  Findlay  and  Lambert  (1989;  as  cited  in  McFarlane 
&  Latorella,  2002)  reported  that  reminders  are  a  useful  technique  in  recovering  from 
interruption.  Another  design  technique  to  enhance  performance  of  responding  to  the  alert  is 
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when  information  (e.g.,  numbers)  is  presented  in  the  same  location  on  the  screen,  rather  than  in 
dispersed  areas. 

•  Negotiated  interruption  is  that  which  is  controlled  by  the  human.  Woods  (1995,  as  cited  in 
McFarlane  &  Latorella,  2002)  hypothesized  that  humans  are  better  than  computers  when  it 
comes  to  diverting  attention.  Thus,  Woods  proposes  that  alerts  should  be  subtle  enough  to  let 
the  human  decide  when  to  direct  their  attention  to  a  secondary  task.  For  instance,  displaying 
alerts  separately  from  the  primary  task,  but  in  a  visible  way,  allows  users  to  attend  to  the  task  if 
they  choose,  or  ignore  it  (Lieberman,  1997;  as  cited  in  McFarlane  &  Latorella,  2002).  Oberg 
and  Notkin  (1992;  as  cited  in  McFarlane  &  Latorella,  2002)  created  a  system  design  where  the 
alert  would  pop  up  near  the  operator’s  curser  position.  Alerts  were  also  colour  coded  so  that  the 
older  an  alert  was,  the  more  saturated  in  colour  it  appeared;  urgent  alerts  changed  darker  faster 
than  non-urgent  alerts.  Although  this  system  was  not  compared  to  other  designs,  anecdotes 
attest  to  the  system’s  usefulness.  Shneiderman  (1992;  as  cited  in  McFarlane  &  Latorella,  2002) 
listed  various  techniques  to  obtain  user  attention,  namely  intensity,  marking,  size,  choice  of 
fonts,  inverse  video,  blinking,  colour,  colour  blinking,  and  audio. 

•  Mediated  interruption  gives  control  over  the  interruption  to  a  third  source,  or  mediator  (e.g.,  a 
personal  digital  assistant,  an  answering  machine,  etc).  Czerwinski,  Cutrell  and  Horvitz  (2000; 
as  cited  in  McFarlane  &  Latorella,  2002)  suggested  that  the  system  should  queue  alerts  until 
the  user  has  a  natural  break.  Similar  to  this,  a  program  that  can  predict  what  the  user  would  do 
next,  can  interrupt  with  relevant  information  at  the  appropriate  time  (e.g.,  Hammer  &  Small, 
1995;  as  cited  in  McFarlane  &  Latorella,  2002). 

•  Scheduled  interruption  can  be  thought  of  as  a  predetermined  time  where  an  operator  allows  for 
distractions.  Alerts  to  anomalous  information  received  from  regularly  scheduled  Maritime 
Patrol  Aircraft  (MPA)  flights  are  examples  of  interruptions  that  may  be  most  relevant  to  the 
RJOC  operators. 

Toet  (2006)  also  identified  some  less  intrusive  methods  for  presenting  alert  warnings  in  his  review 
of  the  literature  on  gaze  direction  tracking.  The  author  suggests  that  an  alert  system  that  is  informed 
of  the  operator’s  gaze  direction  will  be  able  to  present  information  to  that  operator  in  a  way  that 
will  maximize  responsiveness.  Specifically,  the  interface  can  reduce  visual  clutter,  enhance  the 
operator’s  attentive  capacity  and  direct  the  operator’s  attention.  For  the  computer  to  perform  the 
action  of  eye  tracking,  a  non-intrusive  video-based  tracking  system  must  be  installed  to  monitor  the 
operator’s  gaze  and  direct  attention. 

Of  particular  interest  for  this  review  are  the  techniques  used  to  present  visual  information  on  the 
display  screen.  These  techniques  include  non-distortion,  distortion  and  gaze  contingent  techniques. 

Non-Distortion  Oriented  Techniques 

A  semi-transparency  (multi-layer  displays)  allows  operators  to  quickly  shift  their  attention.  The 
display  shows  two  different  views  (layers),  one  in  the  foreground  (e.g.,  overview)  and  one  in  the 
background  (e.g.,  detailed  map).  Operators  can  shift  their  attention  between  the  two  views  best 
when  views  were  50-70%  transparent. 

Distortion-Oriented  Techniques 

These  techniques  combine  a  detailed  (full  size  or  enlarged)  representation  of  the  regions  of  interest 
with  a  less  detailed  (compressed)  representation  of  the  remaining  regions  to  draw  the  operator’s 
attention  to  the  critical  information.  Studies  have  shown  that  participants  who  use  distortion 
techniques  are  faster  at  navigation  (Gutwin  &  Fedak,  2004;  as  cited  in  Toet,  2006). 
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Gaze  Contingent  Displays 

These  displays  include  multiresolution  displays,  stereoscopic  displays,  and  guiding  displays. 
Multiresolution  and  stereoscopic  displays  track  the  user’s  gaze  and  enhance  the  area  being 
attended.  Attention  guiding  displays,  however,  use  a  non-intrusive  method  to  direct  the  user’s  gaze 
to  critical  information.  For  example,  overlaying  red  dots  on  a  video  clip  will  attract  the  user’s 
attention.  Attentive  interfaces,  such  as  perceptual  intelligent  interfaces,  can  adapt  their  behaviour 
according  to  the  user.  The  interface  tracks  the  operator’s  interactions  over  time,  thus  predicting 
their  future  actions. 

The  distortion,  non-distortion  and  gaze  contingent  displays  can  be  effective  for  non-intrusive 
alerting  systems.  The  semi-transparency  technique  allows  operators  to  quickly  and  easily  shift  their 
attention  between  tasks.  Similar  to  the  fade  technique  described  by  McCrickard  et  al.  (2003b), 
semi-transparent  alerts  that  pop  up  in  the  foreground  of  the  display  draw  attention  without  being 
intrusive.  Attention  guiding  displays  can  be  less  intrusive  methods  for  directing  eye  gaze  to  critical 
information  by  directing  an  operator’s  attention  to  the  information  by  overlapping  red  dots.  Lastly, 
eye  contact  displays  can  prevent  irritating  alerts  by  silencing  once  an  operator  looks  at  them. 

4. 2.2.4. 1.1  Assessment 

Hautamaki  et  al.  (2006)  provide  a  useful  model  for  presenting  alert  information  to  operators  in  a 
timely,  context  sensitive  and  multi-modal  manner.  In  particular,  they  propose  that  alerts  become 
more  intrusive  as  the  severity  of  the  alert  consequences  increase.  For  example,  an  initial  alert  is 
presented  in  the  form  of  text  on  the  operator’s  screen.  If  the  alert  is  not  acknowledged  in  a 
sufficient  time  period,  and  the  condition  persists  or  worsens,  the  alert  is  presented  as  a  flashing  text. 
The  alert  then  proceeds  to  an  audible  alert,  followed  by  the  addition  of  verbal  instructions.  While 
there  may  be  no  comparable  operational  requirements  in  the  RJOCs  that  would  require  temporal 
changes  in  the  alert  to  signify  increased  urgency,  the  example  categories  of  “intrusiveness”  do 
provide  some  specific  design  options  that  may  be  applicable. 

Some  of  the  methods  described  by  McFarlane  and  Latorella  (2002)  could  potentially  be  used  in  a 
less  intrusive  alerting  system  for  example,  making  alerts  subtle  and  presenting  them  in  such  a  way 
that  they  are  visibly  separate  from  the  main  task  but  still  visible  to  the  user  and  providing  coding  of 
alert  priority.  More  questionable  is  the  suggestion  to  present  alert  information  in  the  same  area  of 
the  display  as  the  primary  task,  which  could  potentially  result  in  attention  being  immediately  drawn 
away.  Also  the  accuracy  and  the  reliability  of  the  technology  to  predict  breaks  in  operator  tasking 
or  reduced  workload  remains  unproven,  thus  recommendations  for  design  concepts  based  upon  this 
would  appear  premature. 

The  methods  proposed  in  Toet’s  (2006)  paper  would  have  limited  applicability  to  an  operational 
environment,  where  technology  for  detecting  gaze  direction  cannot  be  realistically  implemented. 
Although  the  multi  layer  or  fish  eye  methods  are  not  compatible  with  existing  constraints  within 
GCCS-M,  the  use  of  semi-transparent  designs  for  an  alerting  system  could  be  feasible. 

Limit  operator  information  overload 

Limiting  information  overload  of  the  operator  is  an  essential  feature  of  a  non-intrusive  alerting 
system.  In  discussion  of  a  soft-desk  control  room,  Dicken  (1999)  proposed  some  ideas  for  limiting 
operator  information  overload.  His  review  was  based  on  the  trend  in  process  plants  to  replace  hard- 
desk  operator  interfaces  (i.e.,  horse  shoe  control  desk  which  includes  dials,  meters,  chart  recorder, 
knobs,  and  buttons)  with  soft-desk  operator  interfaces  (i.e.,  computer-based  Visual  Display  Units 
with  management  software  and  user  displays).  In  the  past,  plant  operators  were  typically  in  charge 
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and  ‘drove’  the  plant;  however  with  increasing  automation,  the  operator  has  evolved  from  driving 
the  plant  to  being  driven  by  alerts.  With  the  increasing  dependence  on  alert  systems,  unacceptable 
problems  such  as  generation  rates,  nuisance  alerts,  and  poor  performance  during  alert  floods  have 
arisen.  Consequently,  Dicken  (1999)  provided  a  careful  look  at  important  design  concepts  that 
should  be  considered  when  switching  from  hard-  to  soft-desk  alert  system  facilities.  First,  he  noted 
that  alert  lists  are  prime  operating  tools  for  soft-desk  systems  (see  Section  4.2.5  Manage  Alerts  for 
detailed  information).  The  system  indicates  an  alert  by  flashing  one  of  three  colours,  which 
corresponds  to  urgency  and  matches  the  list  of  alerts.  To  prevent  visual  overload,  alerts  are  grouped 
and  are  attached  to  the  same  plant  item  icon.  For  example,  alerts  relating  to  a  single  mill  are 
grouped  together. 

Grootjen,  Bierman  and  Neerincx  (2006)  also  proposed  some  ideas  for  limiting  operator  information 
overload.  Over  multiple  projects,  Grootjen  et  al.  (2006)  identified  problems  in  process  control  such 
as  information  type,  information  volume,  task  integration,  increased  autonomy,  increased 
complexity,  low  personal  costs,  low  training  costs  and  legislative  constraints  (for  more  details,  see 
Grootjen  et  al.,  2006).  To  address  these  problems,  Grootjen  et  al.  (2006)  designed  an  interface  to 
optimize  the  operator’s  cognitive  task  load  (CTL)  by  transferring  a  task  or  part  of  a  task  to  another 
person.  This  adaptive  user  interface  was  designed  to: 

•  Show  only  the  categories  with  active  alerts  (empty  categories  are  not  shown).  The  interface 

contains  the  operator’s  alerts  and  the  alerts  of  other  operators; 

•  Show  only  the  buttons  that  are  relevant  for  the  alerts  the  operator  handles; 

•  Provide  operators  with  an  icon  that  allows  them  to  redirect  alerts  to  other  operators. 

Grootjen  et  al.  (2006)  evaluated  the  effectiveness  of  the  interface.  While  participants  worked  in 
pairs  to  solve  problems,  they  were  presented  with  a  number  of  different  alerts.  Participants  received 
no  task  allocation  (TA)  support,  task  allocation  advice  from  the  system,  or  a  notice  that  the  system 
had  reallocated  the  task.  Overall,  Grootjen  et  al.’s  (2006)  adaptive  interface  was  rated  positively. 
Participants  reported  that  the  allocation  interface  was  pleasant  to  use,  not  difficult,  useful,  and 
allowed  them  to  solve  problems  faster  and  better.  The  reallocation  of  alerts  was  not  found  to 
require  a  lot  of  effort  or  be  confusing.  For  the  automatic  alert  group,  automatic  TA  of  alerts  was 
found  to  moderately  disturb  their  normal  way  of  working  and  was  rated  as  moderately  annoying. 
For  the  advice  group,  the  TA  advice  was  not  reported  to  disturb  their  normal  way  of  working. 

4. 2. 2. 4. 1.2  Assessment 

With  respect  to  alert  intrusiveness,  Grootjen  et  al.’s  (2006)  adaptive  interface  offers  some  useful 
functions.  However,  the  focus  of  the  work  is  primarily  on  process  control  types  of  tasks  and  their 
associated  interfaces,  which  have  little  direct  comparability  to  RJOC  operators  working  with 
GCCS-M. 

Maintain  situation  awareness  of  primary  task 

Maintaining  or  enhancing  situation  awareness  of  the  primary  task  is  another  important  design 
feature  of  an  alert  warning  system.  McFarlane  and  Berger  (2004)  were  concerned  with  an 
operator’s  ability  to  maintain  situation  awareness  while  using  notification  systems.  Automatic 
notification  systems  constantly  monitor  and  generate  alerts,  but  these  alerts  often  interrupt  other 
activities.  Although  people  do  not  generally  perform  sustained,  simultaneous,  multi-channel 
sampling  well  on  their  own,  they  can  do  so  when  provided  with  specific  interface  support.  An  alert- 
based  information  stream  can  deliver  tasks  and  information  to  support  the  operator’s  ability  to  a) 
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constantly  monitor  a  dynamic  environment,  b)  collaborate  and  communicate  with  other  people  in 
the  system,  and  c)  supervise  background  autonomous  services.  The  Human  Alerting  and 
Interruption  Logistics  (HAIL)  technology  was  developed  to  improve  an  operator’s  ability  to 
maintain  situation  awareness  during  high  rates  of  alerting.  The  HAIL  user  interface  is  designed  to 
improve  an  operator’s  ability  to  process  alerts  and  to  reduce  the  number  of  interruptions  during 
complex,  stressful  tactical  situations.  Compared  to  the  Identification  Supervisor  (IS)  operator  for 
the  Aegis  Weapon  System,  HAIL  is  said  to  reduce  the  number  of  operator  interruptions,  improve 
operator  situation  awareness  for  each  alert  and  status  information,  improve  control  of  alerts 
requiring  action  responses,  and  assist  in  returning  to  the  operator’s  original  task.  For  an  illustration 
of  the  current  Aegis  alert  processing  system  and  the  HAIL-enhanced  Aegis  processing  system,  see 
McFarlane  and  Berger  (2004). 

Rather  than  displaying  alerts  in  a  single  window,  the  HAIL  system  pre-processes  alerts  and 
displays  them  in  the  appropriate  window.  Operators  are  then  able  to  negotiate  their  response  to  the 
alert  by  choosing  to: 

•  Surface  the  alert  (i.e.,  make  it  visible); 

•  Defer  alert  and  surface  next  alert; 

•  Complete  alert  and  surface  next  alert; 

•  Defer  alert;  or 

•  Complete  alert. 

McFarlane  and  Berger  (2004)  tested  the  effectiveness  of  HAIL  with  experienced  naval  operators  in 
an  operator  simulation  using  an  Aegis  alerting  system  and  a  HAIL-enhanced  Aegis  alerting  system. 
In  general,  results  of  the  HAIL  technology  were  positive.  McFarlane  and  Berger  (2004)  conclude 
that  HAIL  increases  warfighter  performance  by  providing  operators  immunity  to  the  effects  of 
trash  alerts,  fewer  interruptions,  better  alert  situation  awareness,  and  easier  recovery  of  non-alert 
work  after  handling  an  alert.  Operators  reported  that  HAIL  made  it  easier  for  them  to  distinguish 
between  noise  alerts  and  important  alerts. 

4. 2. 2. 4. 1.3  Assessment 

This  paper  provides  some  relevant  concepts  for  providing  operators  with  separate  functionality  for 
alert  actions,  alert  information  and  alert  management.  One  caution  to  be  remembered  is  that  this 
system  was  designed  for  more  dynamic  tactical  maritime  displays  than  is  the  case  with  GCCS-M, 
where  data  is  updated  at  a  slower  rate  (i.e.,  more  time-late  data)  and  tactical  decisions  and 
responses  do  not  have  to  be  made  with  battlefield  urgency. 

Alert  reduction 

Ahnlund,  Berguist  and  Spaanenburg  (2003)  argue  that  many  alerts  are  distractive  and  do  not  alert 
the  operator  to  important  information.  One  way  to  potentially  reduce  the  intrusiveness  of  alerts  and 
their  impact  on  primary  task  performance  is  to  reduce  the  number  of  alerts  to  which  operators  must 
attend.  In  particular,  nuisance  alerts  are  problematic  because  they  unnecessarily  overload  operators 
with  alerts,  which  increases  operator  workload  and  has  been  known  to  cause  operators  to  turn  off 
alert  systems  altogether  (Sorkin,  1988).  Ahnlund,  Berguist  and  Spaanenburg  (2003)  designed  an 
alarm  cleanup  methodology  and  computerized  tool  to  remove  nuisance  alerts  from  user  interfaces 
that  would  not  interfere  with  overall  operations. 
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The  alarm  cleanup  method  uses  a  software  program  called  the  Alarm  Cleanup  Toolbox  (ACT). 
ACT  helps  tune  alert  limits  and  develops  algorithms  to  reduce  the  number  of  alerts.  To  perform  an 
alarm  clean  up,  the  following  steps  are  applied: 

•  “Extract  signal  data  during  normal  operation.  This  is  the  difficult  part  since  most  control 
systems  are  installed  without  a  logging  device. 

•  Extract  information  about  the  signals,  such  as  the  current  alert  limits  and  the  applied  signal 
processing  methods. 

•  Examine  the  control  system’s  built-in  functions  and  programmable  capabilities. 

•  Perform  an  off-line  analysis  of  the  signals  using  ACT. 

•  Discuss  and  validate  the  suggested  alert  reduction  methods  and  implementation  decisions 
with  the  operators  and  personnel  with  process  knowledge. 

•  Implement  the  discussed  improvements  into  the  control  system”  (p.  8). 

To  validate  the  ACT  and  alarm  clean  up  method,  researchers  implemented  this  technology  at  a  bio¬ 
fuelled  District  Heating  Plant  (FFC).  They  were  able  to  track  every  alert  to  determine  if  the  alert 
was  removed  or  delayed.  Results  showed  there  were  83%  fewer  alerts  while  using  ACT. 

Chyssler,  Burschka,  Semling,  Lingvall  and  Burbeck  (2004)  were  also  interested  in  alert  and  false 
alarm  reduction,  specifically  within  the  security  field.  With  respect  to  security,  issues  such  as  alert 
reduction,  false  alarm  reduction,  information  correlation,  and  preventable  total  service  collapse 
need  to  be  considered.  Chyssler  et  al.  (2004)  examined  alert  and  false  alarm  reduction  through 
improving  the  quality  of  Intrusion  Detection  Systems  (IDS).  According  to  Chyssler  et  al.  (2004), 
alerts  can  be  interpreted  by  their  severity,  number,  frequency,  variety,  uniqueness  and  payload.  The 
authors  present  an  architecture  that  incorporates  IDS. 

The  IDS  performs  the  following  tasks: 

•  Static  filtering:  A  system  can  produce  many  irrelevant  messages,  including  false  alarms. 
Tuning  a  Large  Complex  Critical  Infrastructure  (LCCI)  is  dependent  on  a  static  network. 
Known  problems  in  the  system  are  handled  by  static  filters.  These  filters  omit  irrelevant 
data  either  by  ignoring  or  deleting  the  alert.  Deleted  alerts  are  permanently  removed  from 
the  system.  Ignored  alerts  are  saved,  but  not  forwarded  to  the  next  agent. 

•  Adaptive  filtering:  Unknown  problems  in  the  system  are  handled  by  adaptive  filters.  These 
filters  categorize  messages  as  interesting  or  uninteresting. 

•  Aggregation:  Aggregation  is  the  combining  of  frequent  alerts  into  one  alert.  This  reduces 
the  operator’s  workload  and  lessens  the  intrusiveness  of  the  alerts. 

•  Correlation:  The  correlation  agent  analyzes  the  data  and  determines  whether  the  alert  is 
interesting  or  uninteresting  by  comparing  the  sum  to  a  determined  threshold. 

Chyssler  et  al.  (2004)  applied  this  architecture  to  a  realistic  Internet  Protocol  (IP)  environment 
where  the  network  experienced  internet  attacks.  Results  showed  that  static  filtering  reduced  the 
frequency  of  alerts  and  that  messages  were  combined  into  one  alert  when  there  was  more  than  a 
65%  similarity. 

Overall,  the  method  described  by  Chyssler  et  al.  (2004)  shows  promising  results  to  reduce  the 
number  of  alerts  and  false  alarms  experienced  by  operators.  Although  the  study  is  in  the  context  of 
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computer  networks  and  internet  attacks,  it  can  be  applied  to  the  maritime  domain.  Static  filtering 
(i.e.,  ignoring  or  deleting  alerts),  adaptive  filtering  (i.e.,  classifying  alerts  as  interesting  or 
uninteresting),  aggregation  (i.e.,  combining  frequent  alerts  into  one  alert)  and  correlation  (i.e., 
determining  if  the  vector  is  interesting  or  uninteresting  using  algorithms)  data  show  that  when 
implemented,  these  techniques  can  greatly  reduce  the  number  of  alerts  and  false  alarms. 

4. 2.2.4. 1.4  Assessment 

The  focus  of  these  papers  is  the  reduction  in  nuisance  and  other  non-critical  alerts  through  smart 
technology.  As  such,  these  papers  will  be  of  more  interest  to  those  who  will  be  responsible  for  the 
development  of  the  technology  and  algorithms  that  will  determine  which  data  anomaly  conditions 
in  the  RMP  will  merit  being  brought  to  the  attention  of  operators.,  They  do  not  apply  directly  to  the 
current  project  where  the  focus  is  on  interface  design  approaches  to  minimise  alert  intrusiveness. 

4. 2.2.5  Summary 

The  following  table  summarizes  some  of  the  key  recommendations  and  design  principles  from  the 
literature  on  alert  warnings  or  indicators  that  are  potentially  relevant  to  the  design  and 
implementation  of  initial  alert  warnings  for  the  RJOC  operators. 


Table  12:  Summary 


Form  of  literature 

Issue 

Recommendations  or  design  principles 

Models 

Alert  display 

Define  alerts  by  colour  and  shape 

Low  interruption 

Alerts  that  are  low  in  interruption  are  the  secondary  display  and  ambient 
media 

Design  Guidelines 

Operator  workload 

Keep  set  points  at  a  level  to  avoid  nuisance  and  false  alarms 

Operators  should  be  able  to  suppress  or  shelve  certain  alerts 

The  rate  at  which  alert  lists  are  populated  must  not  exceed  the  users’ 
information  processing  capabilities 

Operators  should  be  able  to  turn  off  non-critical  alerts 
without  erasing  information 

Alert  display 

Do  not  present  status  indications  through  an  alert  system  display 

Alert  acceptance  should  be  reflected  by  a  change  on  the  visual  display 

Alert  presentation  should  use  specific  sounds/colours/etc.  to  identify  the 
category  of  risk  and  priority 

The  auditory  portion  of  an  alert  should  shut-off  automatically  when  it  no 
longer  provides  useful  information 

Empirical  Research 

Alert  display 

Slow  fade  text 

Voice  warning 

High  urgency=sharp  sound,  medium  urgency=soft  sound 

Predictive  system 

A  system  which  predicts  the  operator’s  actions  to  interrupt  when 
appropriate 
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Form  of  literature 

Issue 

Recommendations  or  design  principles 

Design  Concepts 

Operator  workload 

Allow  operators  the  ability  to  control  alert  responses  through  negotiated 
interruption  or  surfacing  the  alert 

Alert  display 

Use  alert  pop  ups  near  cursor  position 

Colour  code  alerts  by  length  of  time  on  the  screen  with  alerts  getting 
darker  in  colour  the  longer  they  are  on  the  screen  (note:  this  assumes 
certain  display  photometric  properties  that  may  or  may  not  be  applicable 
in  the  RJOCs) 

Use  visual  distortion  techniques  to  draw  operator  attention  to  critical 
information 

Show  only  active  alerts 

4.2.3  Comprehend  alert  information  content 

Whereas  the  previous  section  dealt  with  issues  relating  to  bringing  an  alert  to  the  attention  of  an 
operator,  this  section  examines  relevant  research  on  how  to  represent  the  semantic  information 
associated  with  the  conditions  that  gave  rise  to  the  alert.  That  is,  how  information  should  be 
structured  to  provide  immediate  situation  awareness  of  the  cause  and  conditions  of  the  alert. 
Research  relating  to  alert  information  content  was  in  the  form  of  design  guidelines,  empirical 
research  and  design  concepts. 

4. 2. 3. 1  Design  guidelines 

Miller  (2005)  examined  the  role  of  trust  and  etiquette  in  adaptive  automation.  “Etiquette”  embodies 
unwritten  codes  that  define  the  roles  and  acceptable  behaviour  of  participants  in  a  social  setting. 
Etiquette  rules  define  the  expectations  and  interpretations  of  other  people’s  behaviour.  Therefore, 
etiquette  can  serve  a  role  in  human-computer  interaction.  For  instance,  a  system  using  familiar 
domain  jargon  indicates  that  it  is  experienced  with  the  domain  and  should  be  accorded  the  trust 
reserved  for  those  in  that  domain.  Etiquette  can  also  define  polite  and  rude  behaviours.  Humans  are 
prone  to  interpret  computer  behaviour  similarly  to  human  behaviour  (Reeves  &  Nass,  1996;  as 
cited  in  Miller,  2005),  thus,  systems  are  likely  to  elicit  the  same  negative  or  positive  reactions  if 
they  hold  to  or  defy  established  etiquette.  Therefore,  the  information  content  of  an  alert  should 
comply  with  system  characteristics  and  jargon  with  which  operators  are  familiar. 

Miller  (2005)  defined  the  following  guidelines  for  adaptive  automation.  The  system  needs  to: 

•  Infer  tasks; 

•  Allow  operator’s  to  over  rule  the  system; 

•  Adapt  automation  behaviours;  and 

•  Adapt  information  presentation. 

Adapting  information  presentation  can  be  done  by: 

•  Communicating  about  ongoing  and  future  tasks; 

•  U sing  similar  domain  j argon; 

•  Using  text  format  to  report  context  and  tasks;  and 
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•  Overall,  maintaining  similar  human-to-human  etiquette. 

Assessment 

Miller’s  (2005)  guidelines  are  based  on  a  study  which  used  etiquette  and  gained  a  favourable 
response  from  operators.  The  four  bullet  points  relating  to  adaptive  automation  support  the  notion 
of  having  a  function  in  an  alerting  system  that  allows  local  user  configuration.  The  second  set  of 
bullets  deal  more  with  ensuring  that  the  mode  of  the  communication  is  consistent  with  existing 
operational  concepts,  language  and  procedures.  In  this  case,  we  believe  these  principles  should  also 
be  extended  to  cover  the  maintenance  of  existing  “human  to  system  to  human  etiquette”. 

4. 2. 3. 2  Empirical  research 

The  literature  review  uncovered  one  article  describing  empirical  research  relating  to  alert 
information  content.  Steefkerk,  Esch-Bussemakers  and  Neerincx  (2007)  designed  a  context-aware 
alert  system  and  described  the  system’s  implementation.  Alert  systems  should  direct  the  user  to 
important  and  clear  information  to  allow  for  quick  interpretation  and  reaction.  A  prototype  was 
designed  on  a  personal  digital  assistant  (PDA)  using  visual,  auditory  and  information  condensing 
effects.  The  alert  information  included  in  the  prototype  varied  according  to  the  user’s  workload.  In 
low  workload  situations,  the  full  message  (i.e.,  providing  all  the  information)  was  presented  to  the 
user.  Conversely  when  workload  was  high,  only  a  summary  message  was  presented  to  the  user, 
with  the  exception  of  high  urgency  messages.  Further,  in  order  to  prevent  interruption  of  the  user 
while  attending  to  the  PDA,  an  icon  appeared  when  a  new  message  was  received,  rather  than  the 
full  alert.  This  context-aware  prototype  in  which  only  a  summary  message  is  presented  in  high 
workload  conditions  was  compared  to  a  non-adaptive  prototype  which  alerted  the  user  with  full 
messages  regardless  of  workload. 

Results  showed  that  more  targets  were  remembered  in  the  high  workload  context  for  those  using 
the  adaptive  prototype  than  those  who  used  the  non-adaptive  prototype.  No  significant  differences 
were  found  in  terms  of  the  number  of  correctly  remembered  messages.  Participants  who  used  the 
adaptive  system  reported  the  messages  to  be  less  intrusive  than  those  who  used  the  non-adaptive 
system,  especially  during  high  workload  conditions.  Adaptive  systems  were  also  rated  as  less 
interruptive  and  irritating  than  non-adaptive  systems  and  were  generally  preferred.  Overall,  results 
showed  that  a  system  which  presented  all  the  information  related  to  the  alert  during  periods  of  low 
workload  and  a  summary  of  the  information  during  high  workload  was  preferred. 

Assessment 

This  study  has  limited  applicability  to  the  present  project  in  the  sense  that  it  is  based  on  an 
assumption  that  there  will  be  appropriate  technology  available  to  assess  ongoing  operator 
workload.  However,  the  design  concept  of  providing  summary,  rather  than  detailed,  alert 
information  in  certain  conditions  may  be  applicable  and  will  be  pursued  in  the  design  of  a  maritime 
anomaly  alerting  system  for  RJOC  operators. 

4. 2. 3. 3  Design  concepts 

As  described  previously,  Dicken  (1999)  discussed  the  problems  that  could  arise  when  switching 
from  hard  to  soft-desks.  Dicken  also  reported  how  the  information  related  to  the  alert  should  be 
displayed.  There  are  features  available  to  the  operator  to  limit  the  overload  of  information.  The 
operator  has  the  option  of  retrieving  more  information  regarding  the  alert.  This  comes  in  the  form 
of  a  point  and  click  picklist.  The  picklist  has  a  number  of  options  that  give  operators  the  ability  to 
retrieve  more  information  relevant  to  the  alert,  at  a  time  that  is  convenient  to  them.  The  difference 
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between  a  picklist  and  an  alert  list  is  that  the  former  allows  the  operator  to  progressively  select 
further  layers  of  information  concerning  the  alerting  state  as  the  situation  and  time  available  merit. 

Assessment 

The  merit  of  this  approach  is  that  it  would  minimize  workload  by  allowing  operators  to  have 
control  over  what  information  is  needed  at  certain  times.  Thus,  the  general  concept  that  may  be 
applied  to  an  alert  information  window  is  to  structure  information  hierarchically,  thereby  allowing 
users  progressively  more  detailed  information  at  lower  levels,  which  can  be  simply  accessed  by 
point  and  click  approaches. 

4.2.4  Maintain  primary  task 

This  section  focuses  on  the  operator’s  ability  to  perform  his  or  her  primary  task  while  being 
presented  with  alerts.  Research  suggests  that  there  are  three  features  of  an  alerting  system  that  will 
support  an  operator’s  ability  to  main  their  primary  task:  assessing  the  interruptibility  of  the 
operator,  preparing  an  operator  for  an  upcoming  alert,  and  facilitating  the  operator’s  return  to  the 
primary  task.  This  section  is  therefore  further  subdivided  according  to  these  three  features. 

4. 2.4. 1  Assessment  of  interruptibility 

This  section  describes  literature  relating  to  human  factors  and  the  assessment  of  interruptibility  in 
the  design  of  an  alert  system.  Of  the  four  categories  of  literature,  we  were  only  able  to  find  relevant 
papers  relating  to  generic  design  guidelines. 

Design  guidelines 

The  assessment  of  interruptibility  relates  to  the  impact  an  alert  has  on  the  performance  of  the 
operator.  The  goal  is  to  minimize  interruption  to  maintain  performance  and  situation  awareness. 
The  literature  provides  some  evidence  that  emotional  states  have  a  major  impact  on  perception, 
cognition,  and  motor  processes,  which  in  turn,  impact  performance.  Belief  states  (i.e.,  assessment 
of  the  current  situation)  have  also  been  found  to  impact  decision  making  and  response  selection. 
Given  the  impact  that  affect  and  beliefs  have  on  performance,  the  following  authors  have  looked  at 
how  to  adapt  information  presentation  to  the  individual  belief  states  of  operators. 

The  Affective  and  Belief  Adaptive  Interface  System  (ABAIS)  developed  by  Hudlicka  and 
McNeese  (2002),  was  designed  to  address  individual  differences.  Specifically,  ABAIS  uses  an 
adaptive  methodology  framework  capable  of  adapting  the  user  interface  and  information  to  an 
operator’s  current  affective  state,  key  personality  traits,  situation  specific  beliefs,  and  preferences. 
In  particular,  the  ABAIS  system  architecture  uses  an  adaptive  methodology  consisting  of  the 
following  four  modules: 

1 .  User  State  Assessment:  identifies  the  user’s  affective  state  and  task  relevant  beliefs  (e.g., 
level  of  anxiety).  The  User  State  Assessment  module  receives  information  about  the 
operator  and  task  context  to  identify  the  operator’s  predominant  affective  state  and 
situation-relevant  beliefs. 

2.  Impact  Prediction:  identifies  the  effect  of  operator  state  on  performance  (e.g.,  focus  on 
threatening  stimuli).  The  Impact  Prediction  module  inputs  the  identified  affective  state  and 
operator  beliefs  to  determine,  using  rule-based  reasoning,  the  impact  on  task  performance. 

3.  Strategy  Selection:  selects  a  compensatory  strategy  (e.g.,  presentation  of  additional 
information  to  reduce  ambiguity).  The  Strategy  Selection  module  receives  the  predicted 
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impact  as  input  and  selects  a  compensatory  strategy  to  counteract  resulting  performance 
biases. 

4.  GUI/Decision  Support  System  (DSS)  Adaptation:  modifies  the  user  interface  content  and 
format  to  improve  detection,  recognition,  and  assimilation  of  incoming  data  to  enhance 
situation  awareness.  The  GUI/DSS  module  implements  a  selected  compensatory  strategy  in 
terms  of  specific  GUI  modifications.  GUI  modifications  are  based  on  individual  user 
preferences  for  information  presentation  (e.g.,  blinking,  colour  change,  size  change  of 
relevant  display  or  icon). 

Prior  to  using  ABAIS,  each  operator  must  provide  background  information  (e.g.,  individual  history 
information,  baseline  physiological  or  diagnostic  data).  Categories  of  information  include 
personality,  skill,  individual  history  and  adaptation  preferences.  The  information  is  then  used  to 
perform  a  Cognitive,  Affective,  Personality  Task  Analysis  (CAPTA)  to  produce  a  comprehensive 
description  of  possible  behaviours  and  behaviour  states  associated  with  specific  user  states,  traits 
and  beliefs. 

Once  the  user  affective  and  belief  states  are  identified  and  their  likely  impact  is  predicated,  ABAIS 
identifies  a  compensatory  strategy  and  selects  a  means  of  implementing  this  strategy  in  terms  of 
specific  user  interface  modifications.  This  strategy  is  defined  based  on  stable  performance  biases 
and  the  current  task  context.  Once  the  compensatory  strategy  has  been  identified,  ABAIS 
implements  this  strategy  in  terms  of  specific  modifications  to  the  user’s  interface.  Modifications  of 
the  interface  to  present  the  information  can  be  made  by: 

•  Modifying  the  GUI  icons  in  terms  of  attributes  (e.g.,  changing  colour  or  size)  or  modify  the 
icon  appearance  itself. 

•  Modifying  the  display  as  a  whole  by  changing  size,  location,  appearance,  or  contents. 

•  Implementing  changes  to  the  GUI  as  a  whole  or  insert  additional  display  elements  designed 
to  focus  attention  on  particular  areas.  For  example,  reconfigure  entire  set  of  instruments  to 
reflect  a  different  system  model  and  insert  attention-capturing  and  attention-directing 
elements  designed  to  direct  the  user’s  attention  to  a  particular  icon  or  display. 

•  Inserting  new  or  modifying  existing  alert  and  alert  notifications,  or  adding  an  icon  to  a 
display  to  represent  new  information.  An  example  of  a  notification  level  adaptation 
includes  adding  text  regarding  desired  focus  of  attention,  or  adding  an  icon  to  a  display  to 
represent  new  information. 

The  ABAIS  technology  could  prove  very  useful  in  reducing  alert  intrusiveness.  Prior  to  presenting 
information  to  an  operator,  the  system  takes  into  consideration  the  operator’s  stable  beliefs,  current 
affective  state,  and  information  presentation  preferences.  Combining  this  information  potentially 
provides  users  with  information  in  the  most  effective  and  least  intrusive  manner.  We  use  the  term 
“potentially”  because  the  ABAIS  has  not  yet  been  empirically  tested  with  operators. 

4.2.4. 1.1.1  Assessment 

The  fundamental  assumption  of  this  paper  is  that  affective  states  can  be  reliably  and  accurately 
assessed  and  modelled  to  provide  clear  parameters  for  selecting  appropriate  alert  configurations 
and  parameters.  This  approach  is  therefore  highly  speculative  and  subject  to  potentially  serious 
operational  consequences,  if  not  implemented  with  a  very  high  level  of  accuracy  and  reliability. 
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Other  than  gross  examples,  the  paper  does  not  provide  any  insight  or  concepts  on  exactly  how 
different  emotional  or  other  states  would  be  mapped  onto  specific  alert  designs. 

4 . 2  A.  2  Warning  of  impending  alert 

This  section  describes  the  literature  related  to  how  to  prepare  operators  for  an  impending  alert  by 
providing  an  advanced  warning.  Of  the  four  categories  of  papers,  only  empirical  research  was 
found. 


Empirical  research 

Hwang,  Lin,  Liang,  Yenn  and  Hsu  (2008)  developed  a  pre-alert  system  that  would  reduce  the 
frequency  of  alerts  and  examined  whether  these  pre-alerts  should  be  a  text  or  a  graphic.  Reducing 
the  numbers  of  alerts  is  done  by  aiding  the  operator  in  identifying  faults  before  the  alert  is 
activated.  This  method  has  been  applied  to  many  industries  and  has  shown  positive  results  in  power 
plant  control  rooms.  Hwang  et  al.  (2008)  designed  the  pre-alert  system  based  on  the  following  5 
rules  in  deciding  whether  or  not  a  system  is  out  of  control: 

1 .  “Any  one  point  falls  outside  the  upper  control  limits  (UCL)  or  lower  control  limits  (LCL); 

2.  Seven  points  in  a  row  are  continually  increasing  (or  decreasing); 

3.  Cyclical  patterns  of  points  occur; 

4.  Two  out  of  three  consecutive  points  fall  beyond  the  two  sigma  limit; 

5.  A  run  of  five  points  falls  beyond  the  one-sigma  limit  (Aft,  1998)”  (as  cited  in  Hwang  et  al., 

2008,  p.  2). 

The  pre-alerts  were  in  the  form  of  a  text  or  graphic.  The  text  alert  turned  black  to  yellow  and  was 
accompanied  by  a  “ding”  sound  when  the  value  changed  7  times  (dropped  or  rose  consecutively; 
Rule  2).  This  text  changed  from  yellow  to  red  and  was  accompanied  with  an  alert  when  the  value 
was  outside  the  parameters  (Rule  1).  The  other  type  of  pre-alert  was  graphical  which  was  designed 
to  provide  trend  information  of  the  situation.  The  original  alert  did  not  include  the  pre-alert 
function. 

Twenty-six  graduate  students  and  staff  of  National  Tsing  Hua  University  were  randomly  assigned 
to  13  groups  which  included  a  reactor  operator  (RO),  assistant  reactor  operator  (ARO)  and  a 
supervisor.  The  primary  task  of  the  RO  and  ARO  was  to  monitor  6  critical  parameters  (vessel 
pressure,  level,  reactor  feedwater  pump  turbine  vibration,  discharge  pressure,  turbine  vibration  or 
generator  vibration).  The  RO  monitored  the  parameters  of  the  core  flow  and  power  during  the  12 
minute  shutdown  (normal  state).  During  shutdown,  the  ARO  closed  and  opened  valves  and  pumps. 
During  the  load  rejection  (abnormal  state),  5  alerts  went  off.  The  RO  had  to  search  for  10  items  and 
find  solutions  in  10  minutes.  The  alerted  task  was  deciding  if  the  presented  values  were  greater 
than  or  less  than  each  other.  Participants  then  filled  out  a  questionnaire  assessing  their  perceived 
mental  workload. 

Overall  results  showed  that  the  pre-alert  type  significantly  reduced  the  number  of  alerts  and  the 
mental  workload  of  the  operator  and  maintained  situation  awareness.  There  were  no  significant 
differences  found  between  the  text  and  graphic  pre-alert  types  for  reducing  the  number  of  alerts. 
However,  for  the  alerted  task,  operators  had  significantly  more  correct  answers  when  deciding  if 
the  presented  numbers  were  greater  or  less  than  each  other  when  the  alert  was  graphic  compared  to 
textual.  The  operator  had  significantly  more  correct  answers  during  shutdown  (normal  state)  than 
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in  load  reduction  (abnormal  state).  Also,  the  ARO  had  significantly  more  correct  answers  than  the 
RO  during  load  reduction  (abnormal  state). 

In  summary,  Hwang  et  al.  (2008)  discovered  that  a  pre-alert  system  would  reduce  the  number  of 
alerts,  thus  reducing  the  intrusiveness  experienced  by  the  operators.  Although  both  types  of  pre¬ 
alert  systems  would  be  a  benefit  to  the  operator,  Hwang  et  al.  (2008)  recommend  the  text  type  pre¬ 
alert  to  be  implemented  in  control  rooms. 

4. 2. 4. 2. 1.1  Assessment 

The  generalisability  of  this  paper  to  the  operations  room  remains  unknown,  since  the  major  thrust 
is  to  reduce  the  number  of  alerts  (presumably  in  a  context  where  this  is  a  serious  problem  for 
operators).  If  the  alert  frequency  is  high,  the  use  of  an  auditory  alert  may  still  result  in  nuisance 
alerts  for  the  operator.  In  addition,  the  use  of  trend  data  to  cue  the  alert  may  not  be  applicable  to  the 
maritime  anomaly  context. 

Note  that  the  paper  by  Mitchell  (1998)  reviewed  in  section  4.2.2. 3  also  has  some  relevance  to  the 
issue  of  interruptibility. 

4.2 .4. 3  Facilitate  return  to  primary  task 

This  section  examines  relevant  research  on  how  to  best  support  an  operator  in  returning  to  his/her 
primary  task  after  attending  to  an  alert.  That  is,  what  should  be  considered  in  the  design  of  an  alert 
system  (which  may  draw  the  operator’s  attention  away  from  the  primary  task)  so  that  the  operator 
can  easily  and  quickly  return  to  his/her  primary  task?  We  were  only  able  to  find  research  in  the 
form  of  design  concepts  rather  than  models,  design  guidelines  or  empirical  research. 

Design  concepts 

After  attending  to  an  alert,  operators  must  typically  return  to  their  primary  task.  This  can  be 
problematic  for  operators  as  interruptions  to  deal  with  alerts  are  associated  with  increased  errors, 
reduced  efficiency,  and  increased  stress  (McFarlane  &  Latorella,  2002).  In  addition,  interruptions 
such  as  attending  to  an  alert  can  reduce  an  operator’s  situation  awareness  of  the  primary  task 
domain.  Operators  may  experience  a  “resumption  lag”  once  they  return  to  a  primary  task  because 
they  then  have  to  work  to  re-acquire  situation  awareness,  retrieve  suspended  task  goals,  and 
perform  required  actions  (Monsel,  2003;  as  cited  in  St.  John,  Smallman  &  Manes,  2005).  Returning 
to  a  primary  task  after  an  interruption  can  be  particularly  difficult  if  the  primary  task  requires  the 
operator  to  monitor  a  screen  and  detect  changes.  As  noted  by  Smallman  and  St.  John  (2005), 
humans  have  remarkable  difficulty  identifying  changes,  which  is  particularly  true  when  operators 
are  distracted  or  interrupted.  The  longer  the  disruption,  the  more  problematic  it  will  be  for 
operators  to  detect  changes  because  their  memory  of  the  state  prior  to  the  alert  will  decay. 
Smallman  and  St.  John  (2003)  argue  that  current  methods  of  displaying  information  only  add  to  an 
operator’s  difficulties  when  returning  to  primary  tasks  because  current  displays  show  information 
in  real  time,  which  require  operators  to  remember  and  mentally  integrate  previous  information  with 
current  information.  This  forces  operators  to  determine  for  themselves  whether  or  not  changes  have 
occurred.  In  order  to  address  the  effects  associated  with  reduced  situation  awareness  (e.g.,  tunnel 
vision,  resumption  lag)  and  to  improve  change  detection  ability,  Smallman  developed  the  Change 
History  Explicit  (CHEX)  human  computer  interface  tool. 

The  CHEX  tool  augments  human  attention  by  detecting  significant  changes  to  a  situation  and 
logging  these  changes  into  a  table,  which  can  be  sorted  and  filtered  by  the  operator  according  to 
specific  variables  (e.g.,  significance,  change  type,  age).  Table  entries  are  linked  back  to  objects  in 
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the  geographical  display.  In  order  to  evaluate  the  usefulness  of  the  CHEX  tool,  Smallman  and  St. 
John  (2003;  St.  John,  Smallman  &  Manes,  2005)  conducted  a  series  of  studies  comparing  the 
CHEX  tool  to  conventional  displays. 

In  the  first  study,  Smallman  and  St.  John  (2003)  had  80  university  students  monitor  a  Geoplot 
containing  a  high  density  of  aircrafts  (40)  or  a  low  density  (13).  The  aircraft  slowly  moved  about 
an  Own-ship  signal,  changing  direction,  speed,  and  turning  on  and  off  their  fire-control  radar 
(FCR).  The  participant’s  task  was  to  identify  aircrafts  that  turned  critically  threatening,  as  quickly 
as  possible.  Aircrafts  on  the  Geoplot  were  assigned  an  “interest  score”  to  reflect  their  potential 
significance  to  the  operator;  ten  interest  points  were  assigned  to  aircraft  that  were  1)  flying  fast,  2) 
flying  towards  own-ship,  and  3)  had  FCR  turned  on.  Once  an  aircraft  had  a  score  of  30,  it  was 
defined  as  a  “critical  aircraft”.  Aircrafts  with  interest  scores  of  10  or  greater  were  shown  in  yellow, 
whereas  those  with  interest  scores  of  less  than  10  were  faded.  Within  each  density  condition, 
participants  used  one  of  four  different  change  awareness  human-computer  interaction  (HCI) 
schemes,  1)  a  baseline  of  the  Geoplot  and  CRO,  2)  baseline  plus  a  static,  chronologically  sorted 
Change  History  Table,  3)  baseline  plus  Change  History  Table  and  red  circle  alerts  around  all 
aircraft  with  changes  (circles  could  be  removed  by  selecting  the  aircraft),  or  4)  baseline  plus  CHEX 
(a  sortable  Change  History  Table  linked  to  the  Geoplot).  Participants  conducted  both  monitoring 
and  reconstruction  tasks.  For  the  monitoring  task,  participants  had  to  indicate  when  an  aircraft 
became  critical  and  how  many  changes  the  aircraft  had  made.  For  the  reconstruction  task, 
participants  performed  mental  arithmetic  for  a  minute  while  the  scenario  continued  to  play  out  of 
view  then  returned  to  the  display  to  indicate  when  an  aircraft  became  critical  and  the  number  of 
changes.  The  authors  found  that  adding  a  Change  History  Table  and  change  alert  circles  improved 
performance  in  terms  of  the  percent  of  correctly  identified  critical  aircrafts,  but  the  greatest 
improvement  in  performance  was  seen  when  the  participants  were  using  CHEX  in  the  high  density 
condition.  Participants  in  the  CHEX  condition  had  an  80%  improvement  in  change  identification 
speed  compared  to  the  baseline  condition. 

In  the  second  study,  St.  John,  Smallman,  and  Manes  (2005)  were  interested  in  evaluating  the  design 
space  of  situation  awareness  recovery  tools  by  comparing  CHEX  against  an  alternative  tool,  Instant 
Replay.  Instant  Replay  allows  operators  to  return  to  a  monitoring  task  after  an  interruption  and 
replay  the  missed  period  at  high  speed  to  quickly  search  for  any  changes  to  the  situation. 
Participants  were  allocated  into  one  of  five  conditions:  Baseline  (map  and  the  aircraft  data  display 
in  the  lower  right  comer  of  the  screen),  Basic  Replay  (allowed  participants  to  restart  the  scenario 
from  the  beginning  of  the  last  intermption),  Explicit  Replay  (automatically  detected  and  marked 
significant  changes  by  adding  small  red  triangles  to  the  aircraft  symbols  and  a  “pop”  sound), 
Explicit  Markers  (removed  the  replay  function  but  kept  the  red  triangles  and  pop  sounds),  and 
CHEX  (included  a  table  that  logged  the  time,  aircraft  identification  number,  and  a  short  description 
of  the  change).  For  a  screenshot  of  the  CHEX  display,  refer  to  St.  John  et  al.  (2005). 

Each  scenario,  contained  aircrafts  moving  slowly  in  the  display,  interruptions,  and  changes. 
Participants  response  times  using  the  CHEX  tool  were  significantly  faster  than  the  other  tools,  and 
57%  faster  than  the  Baseline  condition.  Participants  in  the  CHEX  condition  also  produced  fewer 
misses  and  fewer  errors  than  participants  in  any  of  the  other  conditions.  Participants  in  the  Explicit 
Markers  condition  produced  few  misses,  but  a  high  number  of  errors  and  moderate  response  times. 
Participants  in  the  Baseline  condition  produced  high  miss  rates,  high  error  rates,  and  slow  response 
times.  Participants  in  the  Basic  Replay  condition  had  the  slowest  response  times. 

McFarlane  and  Latorella  (2002)  reviewed  the  tools  that  could  improve  an  operators’  ability  to 
return  to  primary  tasks  by  designing  user  interfaces  to  present  reminders  about  the  existence  and 
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state  of  interrupted  activities.  Marlin  et  al.  (1991;  as  cited  in  McFarlane  &  Latorella,  2002) 
designed  a  user  interface  that  specifically  allowed  users  to  suspend  and  resume  activities. 
Specifically,  this  design  allows  users  to  explicitly  mark  when  an  interruption  occurred,  which 
allows  the  computer  to  generate  appropriate  recovery  support.  Rouncefield  (1994;  as  cited  in 
McFarlane  &  Latorella,  2002)  used  a  similar  method  in  paper-based  offices  by  having  workers 
mark  their  work  context  before  leaving  to  handle  an  interruption.  These  markers  were  found  to 
facilitate  recovery  of  prior  work  contexts  when  people  returned  to  their  prior  tasks.  This  could  be 
implemented  in  a  computer  environment  by  noting  interruptions  on  an  electronic  notepad  that 
constantly  displays  a  list  of  interrupted  activities  (Cypher,  1986;  as  cited  in  McFarlane  &  Latorella, 
2002).  Finally,  Lee  (1992;  as  cited  in  McFarlane  &  Latorella,  2002)  found  that  marking  the 
primary  task  window  with  an  animated  border,  instead  of  a  static  border,  reduced  the  confusion 
about  which  window  was  active  when  operators  resumed  a  task  after  an  interruption. 

4. 2.43.1.1  Assessment 

While  the  problems  associated  with  recovering  situation  awareness  and  quickly  resuming  a  primary 
task  that  was  interrupted  by  an  alert  are  not  the  primary  focus  of  the  present  project,  these  papers 
provide  some  concrete  suggestions  that  could  be  implemented  into  a  future  operator  interface. 

•  Include  a  table  of  significant  recent  changes. 

•  Automatically  highlight  all  changes  to  a  tracked  vessel  when  a  change  is  selected.  When  a 
change  occurs,  a  pop  sounds  and  a  new  row  is  added  to  the  top  of  the  table.  Selecting  a  row 
highlights  the  row  in  yellow,  highlights  the  aircraft  on  the  map  with  a  yellow  circle,  and 
presents  the  aircraft’s  data  in  the  data  display. 

•  Automatically  link  and  highlight  between  the  “Change  Table”  and  the  Geoplot  when  a 
vessel  in  either  display  is  selected.  This  allows  for  faster  critical  vessel  identification 
because  operators  do  not  have  to  search  the  Geoplot  to  find  the  location  of  the  relevant 
vessel. 

•  Including  the  ability  to  sort  the  “change  table”  as  needed  by  the  operator. 

The  following  aids,  while  not  suited  to  the  current  implementation  of  GCCS  or  Concept  of 
Operations  in  the  RJOCs,  may  also  have  merit  in  allowing  operators  to  quickly  regain  situation 
awareness  of  the  primary  task: 

•  Electronically  marking  primary  tasks  before  attending  to  an  alert; 

•  Using  an  electronic  notepad  to  display  interrupted  activities;  and 

•  Mark  interrupted  task  windows  with  an  animated  border  to  allow  for  quick  identification  of 
interrupted  tasks. 

Finally,  it  should  be  noted  that  the  operational  environments  that  the  authors  of  the  above  studies 
had  in  mind  are  characterised  by  multi-tasking  on  a  single  or  multiple  displays,  highly  dynamic 
data  inputs  and  the  often  need  for  a  rapid  operator  response.  For  the  most  part,  these  are  not 
characteristics  that  would  apply  to  the  RJOCs,  except  under  occasional  and  special  circumstances. 

4.2.5  Manage  Alerts 

Although  not  the  primary  focus  of  the  literature  review,  several  papers  were  found  relating  to 
human  factors  and  the  management  of  alerts  in  the  form  of  design  concepts.  Since  these  could  have 
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potential  relevance  for  the  prototype  design  phase  of  the  project,  they  have  been  included  in  the 
review. 

4.2.5.1  Design  concepts 

It  is  becoming  increasingly  important  to  introduce  intelligent  alert  handling  capability  in  order  to 
manage  alert  systems.  Liu  et  al.  (2003)  developed  an  Intelligent  Alarm  Management  System 
(IAMS)  for  suppressing  nuisance  alerts  and  for  providing  advisory  information  to  help  panel 
operators  focus  and  respond  quickly  to  important  alert  information.  IAMS  was  developed  to 
incorporate  special-purpose  algorithms,  process  knowledge,  and  control  system  expertise.  It 
consists  of  a  graphical  user  interface  (GUI),  a  Data  I/O  function,  an  Alarm/Trend/Knowledge 
Database  and  six  sub-blocks: 

1 .  Statistical  analysis:  counts  alert  numbers  in  real  time  for  different  time  periods,  tags, 
message  types,  and  alert  statuses. 

2.  Nuisance  HI/LO  analysis:  analyzes  high  or  low  alerts  and  suppresses  those  that  are 
repeating. 

3.  IOP  (Input  Open)  analysis:  identifies  the  cause  of  input  open  alerts  and  suppresses  those 
that  are  nuisance  alerts. 

4.  Criticality  analysis:  gives  a  criticality  tag  of  very  important,  important,  less  important,  or 
calculation-related  to  each  alert  message. 

5.  Standing  alert  analysis:  shows  standing  alerts,  warns  of  ramping  alerts,  and  resets  standing 
alerts. 

6.  Monitor  &  recover:  shows  changes  to  distributed  control  system  (DCS)  alert  settings  and 
restores  alert  setting  when  the  nuisance  status  is  cleared. 

In  order  to  aid  operators  in  making  appropriate  responses  to  information,  IAMS  provides  operators 
with  advisory  information  that: 

•  Informs  operators  which  alerts  are  emergent  or  critical; 

•  Provides  operators  with  early  warning  for  alerts  that  will  lead  to  violations  of  high-high  or 
low-low  limits; 

•  Provides  online  maintenance  reports;  and 

•  Provides  alert  statistics. 

All  information  is  provided  to  the  operator  through  a  GUI.  The  GUI  displays  guidance  information, 
criticality,  statistics,  and  an  alert  management  overview,  and  allows  operators  to  suppress  nuisance 
IOP  alerts.  The  operator  controls  the  alert  suppression  by  clicking  the  SPR  button  to  enable  and 
the  RST  button  to  disable  the  alert  suppression  function.  The  IAMS  system  also  allows  operators  to 
obtain  a  control  loop  status  report  over  the  last  work  shift  (SFT),  obtain  maintenance  information 
(MTN),  monitor  alert  information  such  as  setting  changes  (MON),  suppress  nuisance  IOP  alerts 
(IOP),  as  well  as  guidance  information  and  alerts  that  have  occurred  (GID).  Operators  also  have 
access  to  all,  calculation-related,  ordinary,  important,  or  critical  alert  messages  (CRIT,  CAL,  ORD, 
IMP,  EMG,  respectively).  Alert  statistics  reports  can  be  easily  generated  for  the  last  10  seconds,  5 
minutes,  hour,  day,  or  a  special  time  period  (SEC,  MIN,  HR,  DAY,  SPE,  respectively). 
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The  IAMS  not  only  allows  operators  to  suppress  nuisance  alerts,  it  allows  them  to  manage  the  non¬ 
nuisance  alerts.  This  alert  management  system  maintains  operator  situation  awareness  by  providing 
the  necessary  information  to  the  operators  at  their  convenience. 

Another  non  intrusive  alert  management  method  mentioned  in  the  literature  is  an  alert  list.  Riveiro, 
Falkman  and  Ziemke  (2008)  and  Dicken  (1999)  used  an  alert  list  in  their  design  of  an  alert  system. 

An  alert  list  is  an  alert  management  component  of  an  alerting  system,  designed  to  manage  alerts 
that  have  been  presented  to  an  operator.  Riveiro  et  al.  (2008)  designed  an  anomaly  detection  system 
which  includes  an  alert  list.  The  interface  display  of  the  anomaly  detection  system  includes  a 
geographical  map,  controls,  detailed  information,  and  the  alert  list  (shown  in  the  bottom  left  hand 
corner).  This  list  is  shown  in  detail  in  Figure  5. 
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Figure  5:  Alert  list  (Riveiro  et  al.,  2008,  p.  6;  ©  2008IEEE) 

When  a  vessel  is  considered  anomalous,  it  is  given  an  identification  number  and  a  coloured  ellipse, 
reflecting  the  probability  of  the  anomaly.  A  card  appears  on  the  alert  list  containing  information 
such  as  the  object  identification  (ID),  coordinates,  probability,  main  reason,  age  of  alert,  and 
delete/report  buttons.  By  pressing  any  of  these  buttons,  the  operator  can  obtain  more  information 
regarding  the  alert.  The  object  ID  is  the  object  identification  number,  which  identifies  each  vessel 
accordingly.  The  object  ID  is  also  highlighted  with  an  urgency  colour  indicating  the  probability  of 
the  anomaly  (red,  orange,  yellow).  The  probability  of  the  alert  identifies  the  probability  that  the 
alert  is  anomalous.  The  position  of  the  ship  is  given  by  x-  and  y-coordinates.  Every  alert  is  time- 
stamped  which  allows  the  operator  to  determine  how  old  the  alert  is  compared  to  the  other  alerts. 
Finally,  each  alert  on  the  list  contains  a  report  and  delete  button  the  operator  can  click  to  perform 
the  intended  action. 

Dicken  (1999)  discussed  an  alert  list  in  the  context  of  a  recent  operator  desk  change,  from  a  hard- 
to  soft-desk.  A  hard-desk  can  be  described  as  a  horse  shoe  control  desk  which  includes  dials, 
meters,  chart  recorder,  knobs,  and  buttons.  It  consists  of  a  back  panel  where  the  indicators  can  be 
found  along  with  alerts.  A  soft-desk  is  a  hard-desk  incorporated  with  Visual  Display  Unit  (VDU) 
screens.  Along  with  this  modernization  of  the  standard  hard  desk,  alert  systems  have  also  been 
changed  to  include  alert  management  software  and  user  displays. 

As  described  by  Dicken  (1999),  the  alert  interface  has  two  options  for  soft-desk  set  up,  1)  the 
standard  VDU  alert  interface,  or  2)  the  alert  display  and  acceptance  interface.  The  latter  option  is 
performed  by  pointing  and  clicking  a  mouse  on  a  standard  screen.  This  option  also  has  an  alert  list 
as  a  secondary  backup  which  can  be  displayed  on  any  screen  and  is  permanently  on  due  to 
necessity.  These  lists  are  limited  to  a  single  chronological  alert  list  and  are  unusable  during 
emergencies.  Recommended  improvements  to  the  alert  list  are  done  by:  1)  showing  alerts  on  pages 
rather  than  scrolling,  2)  keeping  the  alerts  in  the  same  order  (no  shuffling),  3)  adding  new  alerts  to 
the  bottom  of  the  list,  4)  removing  alerts  only  by  operator  action,  5)  having  a  flashing  marker, 
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rather  than  the  alert  text  flashing,  and  6)  prioritizing  alerts  by  colour.  Dicken  (1999)  notes  that 
these  lists  should  be  filtered,  especially  during  an  alert  flood.  Filters  such  as  priority,  category 
(category  rectangles  are  displayed  on  the  bottom  of  the  list),  named  list  (alerts  associated  with  a 
task  are  built  into  a  filter),  modes  (plant  modes,  e.g.,  stable  operation,  start  up),  and  unaccepted 
(alerts  that  have  not  been  accepted)  can  be  useful  to  prevent  operator  overload. 

The  alert  list  also  contains  a  “shelve”  option  for  the  inevitable  nuisance  alerts.  Until  these  are 
serviced  and  fixed,  operators  can  choose  to  ‘shelve’  these  alerts  to  limit  their  intrusiveness.  This 
shelving  option  allows  operators  to  ignore  false  or  nuisance  alerts  that  should  not  be  deleted  and 
should  not  take  time  away  from  the  primary  task.  However,  unlike  ignoring  alerts,  the  shelving 
option  would  not  remind  the  operators  of  the  alert  because  operators  are  still  responsible  for  the 
shelved  alert  list. 

Assessment 

This  paper  provides  a  good  description  of  the  issues  concerning  the  management  of  alerts  and 
potential  design  solutions.  It  shows  that  an  alert  management  system  could  be  one  method  to  ease 
operator  workload  and  maintain  situation  awareness.  An  alert  management  system  can  organize 
alerts  into  a  coherent  list  which  the  operator  can  access  at  any  time  to  obtain  more  information.  The 
suggested  information  contained  in  these  lists  are  vessel  ID  number,  position,  urgency  of  alert,  age 
of  alert,  and  action  buttons  (ignore,  delete,  report,  shelve).  Alerts  on  the  list  should  be  colour  coded 
according  to  urgency  and  this  colour  should  match  the  actual  alert  (e.g.,  red  for  high  urgency, 
orange  for  medium  urgency,  yellow  for  low  urgency).  The  list  should  also  include  a  detailed 
history  of  all  alerts,  including  those  shelved,  deleted  and  reported.  Lastly,  this  list  should  contain  all 
the  information  found  in  the  actual  alert,  including  links  to  pertinent  information. 

There  was  some  lack  of  detail  concerning  the  implementation  of  the  alert  list  in  both  Riveiro  et  al. 
(2008)  and  Dicken  (1999)  although  it  should  be  acknowledged  that  this  was  not  the  main  focus  of 
either  paper.  For  example  the  following  issues  were  not  addressed:  can  an  alert  list  contain  a 
shelved/deleted/reported/ignored  list  or  should  they  be  treated  separately?  Do  the  alert  lists  include 
alerts  that  are  1  hour  old,  1  day  old,  1  week  old,  etc.?  Can  an  alert  system  display  all  the  alerts  in  an 
appropriate  manner  without  looking  cluttered,  or  are  there  a  maximum  number  of  alerts  that  can  be 
listed?  Further,  maritime  operators  may  have  a  large  number  of  alerts  present  over  the  course  of  a 
watch,  which  could  pose  a  problem  in  terms  of  where  and  how  the  alert  list  should  be  displayed.  In 
the  example  provided,  the  alert  list  looks  like  it  could  fit  7  alerts  across  the  screen,  but  the  paper 
does  not  give  any  indication  of  what  happens  to  the  alert  list  when  it  becomes  full. 

In  addition,  it  is  probable  that  operators  would  find  the  alert  list  more  manageable  and  they  would 
have  improved  situation  awareness  if  the  list  were  able  to  be  sorted  by  alert  severity.  These 
questions  and  more  need  to  be  examined  thoroughly  before  the  appropriate  functionality  and  design 
requirements  for  an  operational  context  can  be  determined. 

Finally,  the  paper  assumes  that  operators  can  usefully  comprehend  categories  of  alert  probability, 
which  remains  an  assumption  for  which  current  empirical  experimentation  provides  no  clear 
guidelines. 
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4.3.1  State  of  current  knowledge 

The  following  summary  table  shows  the  number  of  papers  found  and  reviewed  for  each  of  the  main 
functional  aspects  of  an  alert  system  categorised  according  to  paper  type  (see  Table  13). 


Table  13:  Literature  in  the  report 


Configure 

Alert 

parameters 

Receive  Information 
on  Alert  State 

Alert  Information 
Content 

Maintain 
Primary  Task 

Alert 

Management 

Total 

Models 

0 

1 

0 

0 

0 

1 

Guidelines 

0 

6 

1 

0 

0 

7 

Experiments 

0 

7 

1 

1 

0 

9 

Design 

Concepts 

2 

9 

1 

5 

3 

20 

Total 

2 

23 

3 

6 

3 

37 

Overall,  the  largest  subset  of  papers  was  about  design  concepts,  and  the  majority  of  these  focused 
on  how  to  indicate  to  the  user  that  an  alert  or  alarm  had  occurred.  It  was  somewhat  disappointing  to 
find  so  few  papers  that  provided  conceptual  guidance  for  the  design  of  alert  systems  and  how  little 
empirical  research  had  been  done  to  test  the  validity  or  generalisability  of  design  options.  One 
central  theme  was  evident  in  several  papers,  namely,  the  importance  of  designing  alert  systems  to 
minimise  the  operator’s  mental  workload  and  to  reduce  the  potential  for  annoyance. 

While  we  found  some  empirical  research  on  alerts  and  their  potential  impact  on  operator 
performance,  there  is  a  lack  of  research  explicitly  related  to  non-intrusive  alerts.  This  may  be 
because  much  of  the  literature  has  focused  on  the  traditional  problem  of  how  to  make  an  alert 
intrusive  or  salient  enough  to  catch  the  attention  of  the  operator. 

Recent  attention  has  turned  to  the  issue  of  how  a  high  number  of  false  or  nuisance  alarms  can 
degrade  system  and/or  operator  performance.  This  has  resulted  in  a  body  of  research  attempting  to 
decrease  nuisance  and  false  alarms.  Within  this  body  of  research  are  design  guidelines  and 
concepts  that  may  also  be  useful  in  reducing  the  overall  intrusiveness  of  alerts.  To  that  end,  a 
number  of  researchers  (e.g.,  Edworthy  &  Hellier,  2002;  Brown  et  al.,  2002;  Chyssler  et  al.,  2004; 
Ahnlund  et  ah,  2003)  have  recommended  ways  to  reduce  the  number  of  alerts  an  operator  is 
exposed  to  on  a  daily  basis. 

Using  the  ontology  outlined  earlier  in  the  paper  as  a  reference  point,  we  have  extracted  from  the 
literature  a  number  of  design  principles  for  each  of  the  main  alerting  system  components,  as  shown 
in  Table  14. 
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Table  14:  Alert  design  principles 


Alert  system  parameters 

Considerations 

Specific  design 
recommendations 

Comment 

Configure  alert  parameters 

Operator  preferences 
(individual  differences) 

Interrupt  the  operator  according 
to  their  preferences  (i.e., 
colour,  size,  time,  modality, 
movement) 

This  guideline  should 
be  treated  with  caution 
to  ensure  that 
operators  are  not 
provided  with  the 
freedom  to  create  HF 
inappropriate  designs. 

Receive  Information  on 

Alert  State 

Visual 

Colour  -  differ  by  priority  (e.g., 
red=high  urgency, 
orange=moderate  urgency, 
green=low  urgency)  and  length 
of  time  on  screen 

Need  specific 
guidelines  for  time  on 
screen  or  duty  cycle 
for  flashing  alarms. 

Colour  coding 
implemented  in  design 
prototypes. 

Size  of  alert  icon  -  larger  icons 
for  higher  priority  alerts 

Need  specific 
guidelines  on  size  of 
alert  and  how  this 
relates  to  the  display 
size  and  the  size  of 
windows. 

Solid  v.  blinking  icons 

Need  specific 
guidelines  on  blink 
rates 

Change  display  when  alert  has 
been  accepted/actioned  (e.g., 
remove  alert  from  screen, 
visual  marker  to  note  it  has 
been  accepted) 

Pop-up  alert  near  cursor 

This  may  be  too 
intrusive  for  some 
tasks. 

Not  suitable  for  RJOC 
context 

Slow  fade  v.  blast  or  ticker 
alerts 

Need  specifications  on 
the  dynamics  of  the 
fade 

Movement  at  visual  periphery 

May  be  too  general  a 
recommendation 
without  specifying 
primary  tasks  for 
which  this  would  be 
appropriate. 

Not  suitable  for  RJOC 
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Alert  system  parameters 

Considerations 

Specific  design 
recommendations 

Comment 

context 

Shift  display  to  foveal  region 

This  would  have  to  be 
implemented  with 
caution  because  of  the 
potential  to  impact 
adversely  on  the 
situation  awareness  of 
the  primary  task. 

Not  suitable  for  RJOC 
context. 

Display  arrows  pointing  to  alert 

Not  suitable  for  RJOC 
context 

Visual  distortion  techniques 

This  would  have  to  be 
implemented  with 
caution  because  of  the 
potential  to  impact 
adversely  on  the 
situation  awareness  of 
the  primary  task. 

Not  suitable  for  RJOC 
context. 

Auditory  (NOTE:  none 
deemed  suitable  for  RJOC 
context) 

Voice  saying  “Conflict  Conflict” 

Limited  application 

High  urgency=sharp  sound, 
medium  urgency=soft  sounds 

Need  to  define 
frequency  and 
amplitude 

characteristics  more 
specifically. 

Use  specific  sounds  to  identify 
category  of  risk 

Potential  impact  on 
increased  need  for 
training. 

Automatically  shut-off  auditory 
portion  of  alert  when  it  no 
longer  provides  useful 
information 

Other  modalities 

Tactile  and  olfactory 

Tactile  may  be 
suitable  for  warning 
individual  operators 
who  are  away  from 
their  workstation. 

Need  guidelines  and 
research  on  the 
vibrotactile  profile  and 
its  perceived  urgency. 

Olfactory  unsuitable 
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Alert  system  parameters 

Considerations 

Specific  design 
recommendations 

Comment 

for  most 
environments. 

Alert  Information  Content 

Adaptable  to  workload 

High  workload=summarized 
information,  low  workload=full 
information,  high  urgency=full 
information 

Assumes  that  system 
is  able  to  assess 
workload. 

Not  suitable  for  RJOC 
context. 

Direct  operator  /constrain 
information 

Use  picklists. 

Buttons  relating  to  the 
schematics,  control  panel, 
trends,  point  information, 
actions,  procedures,  and 
history  of  the  alert 

Not  suitable  for  RJOC 
context. 

Maintain  situation  awareness 

Ability  to  gain  information  of  a 
vessel  by  clicking  on  the  vessel 
on  the  RMP 

Highly  relevant  to 

RJOC.  Implemented  in 
design  prototypes. 

Maintain  Primary  Task 

Assess  Interruptibility 

The  system  accounts  for  user’s 
beliefs,  affect,  preferences, 
ongoing  task  priorities 

Technology  not  yet 
available  to  ensure  the 
level  of  accuracy 
required  in  estimating 
interruptibility. 

Pre-alert 

Use  text  rather  than  graph 

Not  applicable  to 

RJOC 

Return  to  primary  task 

Include  a  sortable  table  of 
recent  changes  in  the  situation 
while  operator  was  away. 

More  suitable  for  more 
highly  dynamic 
information 

environments  than  the 
RMP. 

Restore  task  window  to  former 
look 

While  generally 
advocating  this 
approach,  we  caution 
that  depending  on  the 
context  there  is 
potential  for  operator 
disorientation  if  the 
picture/RMP  suddenly 
changes  focus/range 
etc  without  operator 
input. 

Marking  primary  task 

Electronically  mark  task  before 
attending  to  an  alert 

Not  applicable  to 

RJOC 

Electronic  notepad  to  record 
interrupted  activities 

Not  applicable  to 

RJOC 

Mark  interrupted  task  window 

Not  applicable  to 
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Alert  system  parameters 

Considerations 

Specific  design 
recommendations 

Comment 

with  an  animated  border 

RJOC 

Alert  Management 

Alert  list 

Provide  sortable  lists  of  alerts 
that  show:  alert  priority,  alert 
context,  time  of  alert  and 
relevant  information  concerning 
contact  details.  Functionality  to 
re-order  and  delete. 

Implemented  in  design 
prototypes 

Provide  detailed  history  of  all 
alerts  (shelved,  deleted  and 
time  actioned) 

Relevant  to  RJOC  but 
not  in  scope  of  present 
work. 

Not  to  exceed  user’s 
information  processing 
capabilities 

Too  general  to  be 
useful! 

Etiquette 

Communicating  about  ongoing 
and  future  tasks 

This  is  more 
applicable  to  highly 
dynamic  task  contexts. 

Using  similar  domain  jargon 

Ensure  that  design 
approaches  are 
consistent  with  RJOC 
CONOPS  and  GCCS 
style. 

Our  general  assessment  is  that  the  above  represents  a  somewhat  piecemeal  and  haphazard 
collection  of  principles  and  guidelines,  as  might  be  expected  since  they  are  an  agglomeration  from 
many  different  papers  with  quite  different  application  environments  and  goals.  Clearly,  there  is  a 
lack  of  a  unified  design  approach  and  associated  recommendations  for  non-intrusive  alerting 
contexts.  However,  the  detailed  guidelines  found  in  Shorrock  et  al  (2002)  and  Han  et  al  (2007) 
provide  a  good  starting  point  for  an  integrated  guidance  document,  which  would  then  need  to  be 
extended  and  made  more  context  relevant  to  the  RJOCs.  In  addition,  there  would  be  a  need  to 
make  this  guidance  more  specific  than  statements  such  as  “alarms  should  signal  the  need  for 
action”,  “alarms  should  be  detected  rapidly. . “alarms  should  not  annoy,  startle  or  distract 
unnecessarily”  etc.  Thus,  while  these  recommendations  are  sound,  there  is  a  lack  of  information  on 
how  they  are  to  be  implemented  as  specific  design  guidelines. 

4.3.2  Gaps  in  the  literature 

In  general,  there  was  a  common  theme  in  the  literature  relating  to  alert  system  design,  namely 
appropriate  ways  to  alert  an  operator  of  the  situation  at  hand.  Capturing  an  operator’s  attention 
requires  shifting  attention  to  the  alert  and  often  times  away  from  the  primary  task,  although  not 
necessarily  for  a  significant  period  of  time.  One  could  argue  that  the  very  nature  of  an  alert  is  to  be 
intrusive.  Although  the  literature  presented  various  methods  to  lessen  the  intrusiveness  of  the  alert 
system,  the  notion  of  an  explicit 4 non-intrusive’  alert  was  not  mentioned.  Further,  the  impetus  for 
designing  less  intrusive  alerts  appears  to  be  minimizing  operator  annoyance  rather  than  cognitive 
demands. 
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As  shown  in  Table  14  in  the  previous  section,  literature  in  the  form  of  models  that  could  be  used  to 
inform  the  design  of  a  maritime  anomaly  alert  system  was  minimal.  It  is  clear  that  the  potential 
impact  of  poorly  designed  alerts  or  alarms  on  operator  performance  and  behaviour  (e.g.,  turning  off 
nuisance  alerts)  has  been  recognized  and  a  number  of  design  concepts  have  resulted.  However, 
normative  models  that  describe  the  relationship  between  alert  system  parameters  and  human 
cognition  do  not  appear  to  exist.  Such  models  would  be  extremely  valuable  in  providing  a 
theoretical  foundation  for  the  design  of  an  alert  system.  At  present,  the  most  suitable  models  are 
generic  cognitive  information  processing  approaches  those  that  focus  on  attention  sharing  and 
resource  competition.  While  not  the  focus  of  the  present  work,  models  of  alert  annoyance  have 
some  relevance  to  the  design  of  auditory  alerts,  although  they  also  may  lack  the  specificity  to 
inform  design  approaches. 

The  majority  of  the  literature  that  was  found  related  to  the  alerting  cue  itself  with  little  focus  on  the 
other  functional  components  of  an  alerting  system.  Although  there  is  a  relation  between  the  alert 
presentation  and  the  alert  information  content,  most  research  focused  on  how  to  present  an  alert  to 
an  operator,  rather  than  the  information  that  should  be  included  in  the  alert,  when  to  present  the 
alert  (interruption),  how  to  configure  the  alerts  and  how  to  manage  old  and  new  alerts. 

The  literature  does  provide,  however,  some  reasonable  guidance  in  reducing  the  frequency  of 
alarms  and  configuring  alarms  to  an  operator’s  local  priorities. 

There  is  a  significant  gap  in  the  literature  when  it  comes  to  concepts  relevant  to  non-intrusive  alert 
design  guidelines.  That  is,  recognition  of  the  need  to  capture  the  operator’s  attention  while  not 
interfering  with  cognitive  processing  of  the  primary  task  was  not  the  impetus  behind  the  majority 
of  the  guidelines.  There  were  a  few  applicable  design  guidelines  that  were  found  relevant  to  the 
alert  state  and  the  alert  information  content  and  these  guided  some  of  the  design  concepts  that  were 
developed. 

Few  articles  investigated  and  compared  different  modalities  of  alerts.  Given  the  pervasiveness  of 
visual  interfaces,  it  is  not  surprising  that  visual  alerts  were  the  focus  of  the  bulk  of  the  literature. 
However,  there  is  research  to  suggest  that  auditory  and  tactile  alerts  may  be  useful  and  perhaps 
even  more  appropriate  than  visual  alerts  in  certain  contexts.  Factors  relating  to  the  intrusiveness  of 
auditory  and  tactile  alerts,  however,  were  beyond  the  scope  of  this  review  and  would  need  to  be 
further  investigated. 

In  conclusion,  there  was  no  single  paper  that  definitively  addressed  the  issue  of  how  to  design  a 
non-intrusive  alerting  system.  Nor,  did  we  find  a  comprehensive  body  of  information  that  could 
provide  specific  answers  on  how  to  scale  the  intrusiveness  of  alerts.  Thus,  the  conclusions  we  have 
reached  concerning  design  for  non-intrusiveness  are  based  upon  relevant  concepts  extracted  from 
the  more  general  alert/alarm  literature.  In  doing  so,  it  should  be  pointed  out  that  many  of  the 
recommendations  lacked  the  specificity  to  inform  design  approaches. 

The  following  table  summarises  our  assessment  of  the  state  of  the  art  of  the  knowledge  base,  and 
what  future  studies  may  need  to  be  done  in  order  to  provide  a  comprehensive  set  of  guidelines  for 
the  implementation  of  non-intrusive  alerts  to  operational  contexts  where  such  an  approach  is 
merited. 
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Table  15:  Summary  of  literature  relevance  and  need  for  research 


Alert  Function 

Comment 

Configure  alert  parameters 

There  are  some  useful  generic  guidelines  available  to  guide  the  design  of 
interfaces  that  would  allow  operators  to  configure  alert  parameters  and 
priorities.  Examples  of  interfaces  for  rapid  selection  of  rules  and 
configuration  are  available. 

Research  question  -  what  is  the  appropriate  number  of  alert  priorities? 

Receive  information  on  alert  state 

While  there  is  abundant  information  on  the  various  ways  an  alert  can  be 
brought  to  the  attention  of  the  operator,  there  is  little  research  on  how  this 
can  be  done  non-intrusively. 

Research  is  needed  to  determine  the  relative  intrusiveness  of  a  range  of 
design  parameters  pertaining  to  the  alert  size,  location  and  dynamics. 

More  refined  information  processing  models  need  to  be  developed  to  allow 
a  better  understanding  of  how  attention  and  intrusiveness  are  related. 

Comprehend  alert  condition 

There  is  a  sold  base  of  information  available  in  the  general  alarm/alert 
literature.  In  an  RMP  context,  there  is  a  need  to  examine  trade-offs  between 
implementing  the  information  within  the  RMP  window  and/or  providing  a 
separate  window. 

Action  the  alert  condition 

Beyond  the  scope  of  the  present  project 

Manage  Alerts 

Some  useful  general  principles  are  found  in  the  literature. 

In  the  context  of  the  RJOC,  analysis  needs  to  be  performed  of  the 
information  requirements  for  an  alert  management  system  for  data 
anomalies.  May  also  need  to  consider  how  this  would  integrate  within  the 
overall  “alert”  system  within  the  centre. 

Maintain  primary  task 

Assessment  of  interruptibility 

The  technology  for  this  is  not  proven  and  has  the  potential  for  creating 
adverse  operator  reactions.  The  human  factors  literature  in  this  area  was 
not  a  priority  for  this  project.  However,  computer  scientists  have  been 
interested  in  this  problem  and  have  developed  some  interesting  models.7 

Warning  of  impending  alerts 

This  literature  is  probably  of  more  relevance  to  more  highly  dynamic 
information  environments  than  the  RJOC. 

Methods  for  return  to  primary  task 

General  principles  for  rapid  regain  of  situation  awareness  are  applicable. 

Research  questions:  should  RMP  be  refocused/ranged  as  before  the  alert 
was  serviced?  Does  changing  RMP  focus  between  primary  task  and 
actioning  the  alert  cause  loss  of  situation  awareness.  Is  there  a  need  to 
alert  operators  to  any  change  in  RMP  status  on  return  from  alert? 

7  See,  for  example,  Gievska,  S.  and  Sibert,  J.  Using  task  context  variables  for  selecting  the  best  timing  for 
interrupting  users.  ACM  International  Conference  Proceedings,  vol  121,  2005. 
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5.  The  Design  Development  Process 

5.1  Development  of  design  concepts 

Based  on  a  review  of  the  literature,  and  bounded  by  the  primary  focus  of  the  project,  we 
concentrated  on  developing  design  ideas  for  two  primary  functional  elements  of  an  alert  system: 

1 .  An  alert  indicator  which  appears  superimposed  upon  the  GCCS  window  and  is  designed  to 
advise  of  new  alerts  non-intrusively;  and 

2.  An  alert  information  window  (AIW)  which  allows  operators  to  obtain  details  on  specific 
alerts  and  to  manage  alert  lists. 

5.1.1  The  alert  indicator 

With  factors  outlined  in  the  previous  section  in  mind,  we  developed  design  concepts  that  map  onto 
three  levels  of  importance  of  the  information  that  triggers  the  alarm.  Our  discussions  with  an  SME 
and  our  own  analysis  of  potential  requirements  suggest  that  three  priority  levels  represent  an 
appropriate  balance  of  information  categories. 

For  each  information  category  (i.e.,  attribute,  movement  and  VOI  related)  we  have  provided 
operational  examples  of  RMP  anomalies  taken  from  Davenport  (2008)  which  have  been  sorted  into 
the  three  priority  levels  by  an  SME.  There  are  four  basic  information  requirements  for  all  alert 
concepts: 

1 .  To  bring  to  the  attention  of  the  operator  that  an  alert  has  occurred; 

2.  To  provide  an  indication  of  the  alert  priority; 

3.  To  provide  a  cumulative  numerical  indication  of  how  many  outstanding  alerts  are  in  the 
system  (i.e.,  have  not  been  processed  or  cleared);  and 

4.  To  update  the  cumulative  alert  indicator  when  an  alert  has  been  cleared. 

There  are  several  coding  schemes  that  could  be  potentially  used  to  visually  indicate  urgency  level, 
including  factors  such  as: 

•  Colour  stereotypes 

•  Colour/luminance  increments  from  the  background 

•  Size  of  the  alerting  stimulus 

•  Rate  of  flashing 

•  Locus  on  the  display 

Considerations  for  the  design  of  a  visual  alert  indicator  for  each  of  the  three  priority  levels  are 
outlined  in  the  next  section. 

In  terms  of  other  modalities  to  indicate  alerts,  auditory  alarms  were  not  considered  as  we  concluded 
that  the  perceptual  deviation  from  the  operator’s  primary  task  (i.e.,  a  visual  monitoring  task)  would 
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be  too  great  and  they  would  therefore  be  too  intrusive.  Further,  it  is  anticipated  that  auditory  alarms 
would  be  disruptive  to  other  operators  in  the  RJOC. 

A  tactile  interface,  on  the  other  hand,  may  be  feasible  in  that  it  can  be  isolated  to  the  individual 
operator  and  therefore  not  disruptive  to  other  operators.  Furthermore,  the  intrusiveness  of  a  tactile 
alert  can  be  altered  to  imply  a  specific  level  of  priority  of  information.  Considerations  for  the 
design  of  a  tactile  alert  indicator  for  each  of  the  three  priority  levels  are  outlined  in  Section  5.1.3. 

5 . 1. 1. 1  Priority  1  alerts 

A  priority  1  alert  is  an  anomaly  that  is  of  critical  significance.  It  will  require  operator  attention  at 
the  earliest  opportunity  and  may  require  temporary  suspension  of  the  primary  task.  The  occurrence 
of  the  alert  should  be  readily  perceivable  while  an  operator  performs  a  primary  task.  The  alert 
should  clearly  signal  that  it  is  of  high  priority.  Some  examples  of  priority  1  anomalies  are: 

•  Grab  and  dash  fishing  -  a  foreign  fishing  boat  moves  from  the  international  zone  to 
Canadian  waters  (where  it  is  forbidden  from  fishing)  for  a  few  hours  just  before  leaving  for 
its  home  port. 

•  Not  heading  for  port  -  a  vessel  is  heading  in  a  direction  where  there  is  no  harbour,  or  is  not 
heading  toward  its  declared  destination.  Cargo  and  Ferry  vessels  always  go  from  one  port 
to  another  port,  and  generally  by  the  shortest  available  route. 

•  Changes  destination  -  a  cargo  ship  changes  course  in  mid-journey  or  possibly  even  reverses 
it’s  heading  and  returning  to  port. 

•  Heading  into  danger  -  a  ship  is  heading  toward  a  natural  obstacle  such  as  ice  or  non- 
navigable  water. 

•  Regulatory  infraction  -  a  ship  enters,  without  permission,  a  regulated  zone  such  as  the 
Northwest  Passage,  where  ships  must  register  their  plans  and  receive  permission  to 
proceed. 

•  Infringing  a  closed  zone  -  a  ship  is  in  a  zone  of  the  ocean  that  is  closed  to  its  type  of 
commercial  activity,  whether  for  environmental,  wildlife  protection,  or  national  security 
reasons. 

5n1n1n2  Priority  2  alerts 

A  priority  2  alert  is  an  anomaly  for  which  the  related  information  is  important  but  can  be  dealt  with 
as  soon  as  primary  task  activity  permits.  The  occurrence  of  the  alert  should  be  readily  perceivable 
while  an  operator  performs  a  primary  task.  The  alert  should  clearly  signal  that  it  is  of  intermediate 
priority.  Some  examples  of  anomalies  are: 

•  Unexplained  high  speed  -  a  ship  that  is  claiming  (e.g.,  in  call-ins  or  on  AIS)  to  be  a  normal 
merchant  ship  suddenly  starts  travelling  at  a  high  speed  more  typical  of  a  passenger  ship  or 
warship. 

•  Speed  too  slow  -  a  Cargo,  Passenger,  or  Ferry  is  observed  going  slowly.  As  these  vessels 
generally  go  as  fast  as  they  safely  can,  it  may  be  an  indicator  of  a  problem. 

•  Loitering  -  a  cargo  ship  stops  outside  of  or  far  from  a  harbour,  or  steams  very  slowly, 
rather  than  proceeding  directly  into  port. 
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•  Outside  historical  route  -  a  ship  that  historically  follows  a  consistent  route,  is  deviating  or 
slowing  down  for  no  apparent  reason. 

•  Outside  shipping  lane  -  a  ship  that  should  be  in  a  shipping  lane  is  instead  travelling  outside 
the  lane.  Ships  approaching  port  enter  a  “Vessel  Traffic  Management”  zone  and  are 
required  to  stay  within  designated  shipping  routes. 

•  Zone  mismatch  to  activity  -  a  ship’s  location  does  not  match  its  claimed  activity,  where 
that  activity  can  only  be  carried  out  in  specific  regions  of  the  sea,  due  either  to  regulations 
or  to  physical  requirements  of  the  activity  itself. 

•  Littoral  rendezvous  -  many  small  crafts  converge  on  a  larger  ship,  and  then  the  small  crafts 
spread  out  at  high  speeds  to  many  different  ports. 

5.1  .1. 3  Priority  3  alerts 

A  priority  3  alert  is  an  anomaly  for  which  the  related  information  is  less  important  and  will  be 
attended  to  as  time  and  resources  permit.  The  occurrence  of  the  alert  should  not  be  as  readily 
perceivable  as  priority  1  and  2  alerts  and  should  not  draw  attention  to  its  occurrence.  It  should  be 
perceivable  only  when  the  operator  needs  to  check  for  alerts.  The  perceptual  properties  of  the  alert 
should  clearly  indicate  that  it  is  of  lowest  priority.  Some  examples  of  priority  3  anomalies  are: 

•  Track  ends  -  a  ship  track  ends  in  mid-ocean.  A  ship  track  will  normally  not  end,  except  at  a 
harbour  or  by  the  ship  leaving  Canadian  waters. 

•  Proximity  to  infrastructure  -  a  ship  approaches  or  loiters  around  Canadian  infrastructure, 
such  as  oil  production  equipment,  sub  sea  pipelines,  communication  cables,  etc. 

Again,  we  want  to  emphasize  that  the  above  does  not  constitute  a  definitive  set  of  anomalies  that 
may  be  of  interest,  nor  is  the  specific  priority  classification  being  recommended  for  adoption. 

These  assumptions  were  made  simply  to  facilitate  the  development  of  design  concepts. 

5.1.2  Visual  design  concepts 

Many  of  the  concepts  to  be  described  have  used  colour  and/or  luminance  coding  as  one  basic 
approach  to  differentiating  priority.  Based  on  existing  population  stereotypes,  the  priority  coding  is 
as  follows: 

•  Priority  1 :  red  sector  of  the  spectrum 

•  Priority  2:  yellow-orange  sector  of  the  spectrum 

•  Priority  3:  unsaturated,  neutral  areas  of  the  spectrum  (e.g.,  grey) 

The  use  of  red  is  considered  acceptable,  even  though  red  is  used  in  GCCS  to  code  VOI,  hostile  and 
suspect  tracks,  since  there  is  unlikely  to  be  any  possible  confusion  because  of  where  the  red  alert  is 
located  and  the  shape  and  context  in  which  it  appears. 

A  second  general  principle  is  to  locate  the  alerting  stimulus  in  the  periphery  of  the  display  towards 
the  right.  We  did  consider  locating  it  on  the  menu  bar  at  the  top  of  the  screen,  but  this  area  is 
already  cluttered  and  we  believed  that  the  spatial  separation  from  the  menu  bar  would  in  fact 
encourage  cognitive  separation,  so  that  typical  menu  intensive  tasks  would  not  be  compromised  by 
the  adjacent  proximity  of  alerts.  Similarly,  the  alerts  themselves  would  be  more  salient  (enough  to 
capture  the  operator’s  attention  without  being  intrusive)  by  being  spatially  separated. 
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Five  different  designs  have  been  created  and  will  be  discussed  in  details  next. 

5.1. 2.1  Design  1:  Cumulative  Total  Indicator 


Figure  6:  Design  1:  Cumulative  Total  Indicator 

The  alert  category  is  indicated  by  colour.  For  priority  1  alerts,  the  number  in  the  red  box  increments 
and  the  box  blinks  at  rate  of  2  Hz.  The  flashing  continues  until  the  operator  acknowledges  the  alert 
by  clicking  on  it.  This  takes  the  operator  to  the  Alert  Information  Window  (AIW).  When  the 
operator  has  finished  processing  the  alert,  either  one  of  two  conditions  exist.  One,  the  alert  has  been 
dealt  with  and  is  no  longer  of  interest,  in  which  case  the  counter  resets  to  n  (number  of  current 
outstanding  alerts)  -1.  Or  two,  the  alert  remains  in  the  system,  and  the  indicator  no  longer  flashes 
and  stays  at  the  current  value,  in  which  case  the  next  alert  would  increment  this  value  and  flash 
until  attended  to  by  the  operator. 

For  priority  2  alerts,  the  number  in  the  yellow  box  increments  and  the  box  blinks  at  an  approximate 
rate  of  .25-. 5  Hz.  The  flashing  continues  until  the  operator  acknowledges  the  alert  by  clicking  on 
it.  The  remaining  functionality  is  the  same  as  a  Priority  1  alert. 

On  a  new  priority  3  alert,  the  number  in  the  grey  box  simply  increments.  When  the  operator  has 
processed  the  alert,  the  number  decreases  to  n-1. 
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5.1. 2.2  Design  2:  Vertical  Cumulative  Indicator 


Figure  7:  Design  2:  Vertical  Cumulative  Indicator 

This  design  concept  is  similar  to  design  1,  except  the  count  of  outstanding  alerts  is  indicated  by  a 
vertical  progress  bar.  The  above  example  show  several  outstanding  alerts  in  all  three  priority 
categories.  A  vertical  scale  provides  an  indication  of  the  number  of  alerts. 

A  priority  1  alert  is  shown  in  this  design  by  the  number  1  blinking  red  at  a  rate  of  2  Hz  and  the 
associated  red  vertical  bar  incrementing  in  height.  The  flashing  of  the  number  1  to  red  continues 
until  the  operator  acknowledges  the  alert  by  clicking  on  it.  This  takes  the  operator  to  the  AIW. 
When  the  operator  has  finished  processing  the  alert,  one  of  two  conditions  will  exist.  The  alert  has 
been  dealt  with  and  is  no  longer  of  interest,  in  which  case  the  vertical  bar  decrements  by  one  unit, 
or  the  alert  remains  in  the  system,  and  the  indicator  bar  stays  at  the  current  height;  in  either  case  the 
box  background  reverts  to  grey.  The  next  alert  would  again  increment  the  height  of  the  bar  and  the 
box  would  flash  red  again  until  attended  to. 

A  priority  2  alert  is  shown  in  this  design  by  the  number  2  in  the  second  box  blinking  yellow  at  an 
approximate  rate  of  .25-. 5  Hz.  and  the  vertical  bar  incrementing  in  height.  The  flashing  continues 
until  the  operator  acknowledges  the  alert  by  clicking  on  it.  The  remaining  functionality  is  the  same 
as  a  Priority  1  alert. 

On  a  new  priority  3  alert,  the  vertical  bar  simply  increments.  When  the  operator  has  processed  the 
alert,  the  number  decreases  to  the  current  outstanding  number  minus  1,  or  remains  the  same  if  the 
alert  is  not  deleted. 
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5. 1.2.3  Design  3:  Horizontal  Indicator  Bar 


Figure  8:  Design  3:  Horizontal  Indicator  Bar 

The  three  alert  categories  are  indicated  along  the  bottom  of  the  display,  segregated  by  position  and 
colour  coding.  It  was  assumed  that  on  the  east  coast  contacts  on  the  left  hand  side  of  the  screen  are 
generally  considered  higher  priority  than  those  on  the  right  side  of  the  screen  simply  because  they 
are  closer  to  land.  For  this  reason  we  suggest  that  priority  1  alerts  are  situated  on  the  left  hand  side 
of  the  screen  for  east  coast  RJOC  operators  while,  for  operators  on  the  west  coast,  high  priority 
alerts  are  positioned  on  the  right  side.  That  is,  the  design  will  be  coast  dependent.  This  assumption 
should  be  validated  with  both  east  and  west  coast  operators. 

A  priority  1  alert  is  indicated  in  this  design  by  the  first  empty  rectangle  in  the  left  most  group 
turning  red  and  blinking  at  a  rate  of  2  Hz.  The  flashing  continues  until  the  operator  acknowledges 
the  alert  by  clicking  on  it.  This  takes  the  operator  to  the  AIW.  Again,  when  the  operator  has 
finished  processing  the  alert,  one  of  two  conditions  will  exist.  The  alert  has  been  dealt  with  and  is 
of  no  longer  interest,  in  which  case  the  box  is  no  longer  filled  with  red,  or  the  alert  remains  in  the 
system,  and  the  box  no  longer  flashes  and  stays  filled.  In  which  case,  a  new  alert  would  result  in 
the  next  horizontal  box  blinking  red  until  attended  to  by  the  operator. 

A  priority  2  alert  is  indicated  in  this  design  by  the  first  empty  rectangle  in  the  middle  group  turning 
light  yellow  and  blinking  at  a  rate  of  2  Hz.  The  flashing  continues  until  the  operator  acknowledges 
the  alert  by  clicking  on  it.  This  takes  the  operator  to  the  AIW.  The  remaining  functionality  is  the 
same  as  a  Priority  1  alert.  As  the  number  of  outstanding  alerts  in  this  category  increases  (seven 
boxes  are  filled),  the  colour  changes  from  light  yellow  to  orange. 

On  a  new  priority  3  alert,  a  grey  box  in  the  right  most  group  is  filled.  When  the  operator  has 
processed  the  alert,  the  fill  is  removed  from  the  box,  or  remains  the  same  if  the  alert  is  not  deleted. 
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As  the  number  of  outstanding  alerts  in  this  category  increases  (seven  boxes  are  filled),  the  colour 
changes  from  light  grey  to  dark  grey. 

5. 1.2.4  Design  4:  Ticker  and  Fading  Bar 


Figure  9:  Design  4:  Ticker  &  Fading  Bar 

This  design  is  similar  to  Design  2  in  that  the  count  of  outstanding  alerts  is  indicated  by  a  vertical 
progress  bar  with  a  scale  to  provide  an  indication  of  the  number  of  alerts  in  the  system  (i.e., 
unaddressed).  In  addition,  individual  incoming  priority  1  and  2  alerts  are  indicated  by  a  bar 
appearing  at  the  bottom  of  the  screen. 

On  a  new  priority  1  alert,  a  red  bar  appears  across  the  bottom  of  the  screen  with  a  message 
scrolling  from  right  to  left  indicating  that  there  is  a  new  priority  1  alert.  A  brief  description  of  the 
type  of  anomaly  is  also  provided  (e.g.,  contact  veering  off-course).  An  unacknowledged  priority  1 
alert  would  also  increment  the  height  of  the  red  bar  in  the  counter  on  the  bottom  right  of  the  screen, 
showing  an  increase  in  the  number  of  active  priority  1  alerts  in  the  system.  The  red  bar  is  present 
and  the  scrolling  continues  until  the  operator  acknowledges  the  alert  by  clicking  it.  The  number  of 
active  priority  1  alerts,  as  indicated  by  the  counter,  would  remain  unchanged  until  individual  alerts 
are  processed. 

On  a  new  priority  2  alert,  an  orange  bar  with  text  indicating  that  there  is  a  priority  2  alert  fades  in 
and  out  of  the  bottom  of  the  screen  at  a  rate  of  5  Hz.  The  bar  and  message  remain  on  the  screen  for 
approximately  2  seconds  before  fading  away  and  then  returning  again  until  the  alert  is 
acknowledged  by  the  operator  (by  clicking  on  it).  An  unacknowledged  priority  2  alert  would  also 
increment  the  height  of  the  orange  bar  in  the  counter  on  the  bottom  right  of  the  screen,  showing  an 
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increase  in  the  number  of  active  priority  1  alerts  in  the  system.  The  number  of  active  priority  2 
alerts,  as  indicated  by  the  counter,  would  remain  unchanged  until  individual  alerts  are  processed. 

On  a  new  priority  3  alert,  the  height  of  the  grey  bar  in  the  counter  at  the  bottom  right  of  the  screen 
would  increment  by  one  indicating  an  increase  in  the  number  of  active  priority  3  alerts.  The 
operator  would  only  notice  this  increment  if  he/she  was  looking  directly  at  the  counter  at  the 
moment  it  increments.  Therefore,  the  operator  would  be  required  to  intentionally  seek  out  active 
priority  3  alerts  rather  than  directly  being  made  aware  of  new  alerts.  The  number  of  active  low 
priority  alerts,  as  indicated  by  the  counter,  would  remain  unchanged  until  individual  alerts  are 
processed. 


5. 1.2. 5  Design  5:  Polygon 


Figure  10:  Design  5:  Polygon 

This  design  is  based  on  principles  of  ecological  interface  design  in  that  it  is  intended  to  represent 
both  the  desired  state  of  the  system  (i.e.,  no  active  alerts  in  the  system)  as  well  as  the  current  state 
of  the  system  (i.e.,  the  presence  of  active  alerts)  in  a  way  that  is  easily  and  quickly  perceived  by  the 
operator.  The  solid  green  triangle  signifies  the  desired  state  (i.e.,  no  active  alerts)  while  the  dotted 
triangle  represents  the  actual  state  (i.e.,  if  there  are  active  alerts  and  if  so,  what  type  priority  of 
alert).  As  alerts  accumulate,  the  dotted  triangle  moves  along  the  axes  for  which  there  are  alerts.  The 
X-axis  (red  in  colour  and  labelled  with  a  “1”)  represents  priority  1  alerts;  the  Y-axis  (orange  in 
colour  and  labelled  with  a  “2”)  represents  priority  2  alerts;  and  the  Z-axis  (coloured  grey  and 
labelled  with  a  “3”)  signifies  priority  3  alerts.  If  there  are  an  equal  number  of  active  priority  1,  2 
and  3  alerts,  the  dotted  triangle  is  an  isosceles  triangle;  if  there  are  unequal  numbers,  the  dotted 
triangle  becomes  skewed  toward  the  priority  level  for  which  there  is  the  most  active  alerts. 
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On  a  new  priority  1  alert,  the  border  of  the  dotted  triangle  skews  toward  the  priority  1  axis  showing 
an  increase  in  the  number  of  high  priority  alerts.  The  dotted  triangle  also  turns  red  and  flashes  at  a 
rate  of  approximately  5  Hz.  The  triangle  remains  red  and  flashing  until  the  operator  acknowledges 
the  alarm  by  clicking  it.  The  number  of  active  priority  1  alerts,  as  indicated  by  the  size  and  shape  of 
the  dotted  triangle,  would  remain  unchanged  until  individual  alerts  are  processed. 

Upon  an  incoming  priority  2  alert,  the  border  of  the  dotted  triangle  skews  toward  the  priority  2  axis 
showing  an  increase  in  the  number  of  medium  priority  alerts.  The  dotted  triangle  also  turns  orange 
and  flashes  at  a  rate  of  approximately  0.5  Hz.  The  triangle  remains  orange  and  flashing  until  the 
operator  acknowledges  the  alert  by  clicking  it.  The  number  of  active  priority  2  alerts,  as  indicated 
by  the  size  and  shape  of  the  dotted  triangle,  would  remain  unchanged  until  individual  alerts  are 
processed. 

On  a  new  priority  3  alert,  the  border  of  the  dotted  triangle  skews  toward  the  priority  3  axis  showing 
an  increase  in  the  number  of  low  priority  alerts.  The  dotted  triangle  also  turns  grey  but  does  not 
flash.  The  operator  would  only  notice  this  increment  if  he/she  was  looking  directly  at  the  display  at 
moment  it  changes  shape  and  colour.  Therefore,  the  operator  would  be  required  to  intentionally 
seek  out  active  priority  3  alerts  rather  than  directly  being  made  aware  of  new  alerts.  The  number  of 
active  priority  3  alerts,  as  indicated  by  the  size  and  shape  of  the  dotted  triangle,  would  remain 
unchanged  until  individual  alerts  are  processed. 

5.1.3  Tactile  design  concept 

This  design  concept  assumes  that  the  RJOC  operator  could  carry  a  pager-type  device  that  would 
transmit  the  alerts.  The  design  is  based  on  a  brief  review  of  literature  on  the  use  of  tactons  for 
mobile  phone  alerts  to  imply  priority  (Brown  &  Kaaresoja,  2006).  Generally,  priority  can  be 
implied  by  the  number,  duration  and  intensity  of  pulses.  That  is,  higher  priority  alerts  would  be 
indicated  by  more,  longer  and  more  intense  pulses. 

On  a  new  priority  1  alert,  the  pager  would  receive  two  pulses  of  approximately  30  milliseconds  in 
duration.  The  intensity  of  the  pulse  would  be  approximately  1.38  V.  The  pulses  would  repeat  until 
the  operator  acknowledges  the  alarm  by  clicking  it.  Interrogating  and  processing  the  alarm  would 
have  to  be  done  on  the  operator’s  computer  workstation. 

On  a  new  priority  2  alert  the  pager  would  receive  one  pulse  of  approximately  10  milliseconds  in 
duration.  The  intensity  of  the  pulse  would  be  approximately  0.98  V.  The  pulse  would  repeat  until 
the  operator  acknowledges  the  alarm  by  clicking  it.  Interrogating  and  processing  the  alarm  would 
have  to  be  done  on  the  operator’s  computer  workstation.  The  operator  would  not  receive  priority  3 
alerts  via  the  pager.  He/she  would  therefore  have  to  be  at  their  workstation  looking  at  the  screen 
and  intentionally  seeking  out  priority  3  alerts. 

5.1 .4  The  alert  information  window 

The  purpose  of  the  Alert  Information  Window  (AIW)  window  is  to  provide  the  operator  with 
specific  information  about  the  alert  details  and  to  manage  alert  lists.  It  is  not  a  window  for  problem 
solving  or  analysis  which  we  assume  will  take  place  using  existing  functionality  in  the  RJOCs. 

For  the  AIW,  only  visual  designs  were  considered.  An  initial  design  for  an  AIW  window  is  shown 
in  Figure  11. 
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Figure  11:  Alert  Information  Window 

The  window  comprises  separate  areas  for  each  alert  priority  with  the  priority  level  colour  coded. 
Within  each  category,  there  is  a  field  for  the  number  of  alerts  in  the  system.  The  information 
provided  includes  track  number,  name  of  the  contact,  the  specific  nature  of  the  anomaly,  and  time 
of  the  report  that  gave  rise  to  the  anomaly. 

The  above  example  shows  what  would  happen  if  the  operator  had  selected  a  priority  1  alert  in  the 
RMP  window.  The  priority  1  section  is  expanded  to  show  all  alerts  and  the  most  recent  alert,  that 
caused  the  alert  indicator  to  flash,  is  highlighted.  It  is  important  to  note  that  the  operator  could 
select  another  alert  in  the  priority  1  list  or  another  alert  category  (in  which  case  that  section  would 
be  automatically  expanded  to  show  all  of  the  tracks  within  that  category).  At  this  point  the  operator 
may  choose  to  refer  back  to  the  RMP  to  see  the  location  of  the  alert  and  the  context  by  clicking  the 
RMP  button  on  the  track  line.  When  this  happens,  the  GCCS  RMP  window  is  brought  back 
showing  the  contact  in  question  highlighted,  as  shown  in  Figure  12  (see  concentric  broken  circle 
around  the  contact  of  interest  in  the  upper  part  of  the  RMP).8 


8  In  the  PowerPoint  presentation  for  the  SMEs,  this  concentric  circle  shrinks  and  expands  dynamically 
around  the  track  symbol  to  better  enable  the  operator  to  locate  the  track  in  question. 


Page  74 


Non-Intmsive  Alert  System 


Humansystems 


CART  EARL  WWIN8QR 


iGECO I 


PNT  309825000 


CAST  PROGRESS 


NEDLLOYD 


KASTNER 


'lykes  LIBERATOR 


COUORIER  06  LJSljfA' 
&JOWAY.  '  *  ■"  ,  CANMAR  GLORjgf'cg  A 

mcke'e' Sons  •  ,1'^  .  »  ,  '  « 

,  .  ACHICHttMAuI  ,  M  SURVE^T  j( 

:  <&°:  i_  '  -*STEELB 

tO^reP  •  <§> 

■»*  ;  yr,  , 


jT|M  •  0  JEANC 

l;-R20PNT^^oi 

~t°  *  *  *  ,  Q  kencmanter  m3S-, 

GR^b  CHARLEVOIX  CAP  BRt  OF  COR  —-WjV  4^7022000*  ^  OVERSEA^, LVERMA 

Q  APRIL  *  x  -  -^'^onveyor  w 

%  TWIjQ  CAPE  SPRY  .CGC 117  ^-Q  L- 1  CANMAR  TRiul 

?G  <•  .  _  -  .^iRB  Cma’b^T’ETCWN  p-1  PNT  304647000  Rio  CAXI 

JWSJ&E,  ©  «r  ra&  M.S.J*. 

iMncffA  m  *J  yv  v*  Pk  LU  mvl1 

m  Ek?,  E 

SsLj.  .  -JB.  ' 


.RASC  MANAS  V  °R''C"A--  '* 

KB  l  MV  CARNIVAL  TRIUMPH*?? — 

PNTC209837 

SrOY.  ANO  BARRY  NO  lEAGl'F  BFAUMONTf— r 
E  a'gl*e'b6sTONt— i  ^ 


Figure  12:  RMP  track  highlight 

When  the  operator  has  finished  analysis  on  the  track  of  interest,  she/he  can  return  back  to  the  AIW 
(by  clicking  on  the  AI  indicator)  and  decide  to  either  leave  the  track  in  the  system  or  to  clear  the 
alert  by  way  of  the  CLEAR  button.  This  would  then  result  in  the  track  being  deleted  from  the  alert 
list,  as  shown  in  Figure  13. 


Figure  13:  Alert  Information  Window  after  a  track  is  deleted 

The  operator  may  choose  to  process  other  alerts  in  the  system,  or  return  to  the  RMP  via  the  button 
DONE  at  the  bottom  of  the  window. 
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5.1.5  The  RMP  pop  up  box 

The  purpose  of  this  box  is  to  provide  the  operator  with  a  shortened  version  of  the  information  about 
the  alert.  The  AIW  provides  full  information  regarding  each  alert  and  allows  operators  to  manage 
all  the  alerts  in  the  system.  The  RMP  pop  up  box  on  the  other  hand,  allows  operators  a  quick  and 
easy  method  to  clear  an  alert  or  leave  it  in  the  system.  The  pop  up  box  appears  when  the  operator 
acknowledges  (i.e.  clicks  on)  the  indicator  for  an  incoming  alert.  This  was  designed  as  a  second 
method  to  present  operators  with  relevant  information.  The  research  team  did  not  come  across  any 
literature  regarding  this  concept.  As  shown  in  Figure  14,  a  pop  up  box  was  designed  to  included  the 
name  of  the  vessel  (e.g.,  Eagle  Boston),  reason  for  the  alert  (e.g.,  Not  heading  to  port),  and  time  of 
alert  (e.g.,  21:15),  as  well  as  action  buttons  (i.e.,  Clear  or  Leave). 


Figure  14:  RMP  pop  up  box 

The  box’s  border  would  be  outlined  in  the  same  colour  as  the  priority  of  the  alert,  in  this  case,  red 
for  priority  1 .  If  the  operator  chose  to  CLEAR  the  alert,  the  system  would  delete  the  alert  and  the 
counter  would  return  back  to  the  previous  number,  in  this  case,  back  to  1 .  If  the  operator  chose  to 
leave  the  alert  in  the  system,  the  counter  would  remain  the  same,  in  this  case,  stay  at  2. 
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6.  Evaluation  and  Review  of  Design 
Concepts  by  Subject  Matter  Experts 

6.1  Preparation  for  Design  Evaluation  and  Review 

Following  the  development  of  the  design  concepts,  a  consultative  process  was  begun  with  the 
Scientific  Authority  to  determine  which  concepts  should  be  taken  forward  for  review  by  Subject 
Matter  Experts  (SMEs).  As  a  result,  it  was  decided  that  the  polygon  design  (because  it  was  thought 
to  non-intuitive)  nor  the  tactile  design  (because  it  was  considered  impractical)  would  be  explored 
further. 

In  preparation  for  the  evaluation,  the  individual  design  elements  (alert  indicator,  alert  information 
window,  alert  track  highlighter  and  RMP  pop  up  box)  were  incorporated  into  a  PowerPoint 
demonstration  concept  that  would  simulate  the  functionality  of  an  alerting  system.  Each 
demonstration  comprised  a  new  alert  initiation  (all  three  priority  levels  considered  sequentially), 
acknowledgment  of  the  alert,  getting  information  on  the  alert  and  clearing  the  alert. 

To  evaluate  the  designs,  questionnaires  were  constructed  to  address  issues  such  as  the  usability  and 
utility  of  the  design,  ease  of  comprehension  and  overall  preferences. 

An  ethical  protocol  was  submitted  to  and  approved  by  the  Human  Research  Ethics  Committee  at 
DRDC  Toronto.  The  approved  ethics  protocol  is  included  in  Annex  A. 

6.2  Method 

The  following  section  outlines  the  methodology  used  in  reviewing  and  evaluating  the  anomaly  alert 
system  design  concepts  with  SMEs. 

6.2.1  Date  and  Location  of  SME  Evaluation 

SME  evaluations  were  conducted  in  the  MAPLE  lab  at  DRDC  Atlantic  from  February  23-26,  2009. 

6.2.2  Participants 

Seven  Subject  Matter  Experts  (SMEs)  were  recruited  to  participate  in  an  assessment  of  the  non- 
intrusive  alerting  designs.  There  were  2  Lt(N),  2  retired  CF  personnel,  2  operators,  and  a  civilian. 
Combined  related  experience  included  a  Common  Operational  Picture  Officer,  Watch  Officer, 
Surveillance  Officers,  Bridge  Watch  Keeper,  and  Surveillance  Database  Operators.  There  were  6 
men  and  1  woman. 

6.2.3  Materials 

Trial  participants  were  presented  with  PowerPoint  representations  of  the  various  design  concepts, 
which  demonstrated  with  animation  how  the  basic  functionality  would  work.  The  concepts  were 
presented  in  the  following  order: 

1 .  Cumulative  total  indicator  with  pop-up  window  (Figure  6) 


\lmnansystems 


Non-Intmsive  Alert  System 


Page  77 


2.  Cumulative  total  indicator  with  AIW  (Figure  6) 

3.  Vertical  cumulative  indicator  with  pop-up  window  (Figure  7) 

4.  Vertical  cumulative  indicator  with  AIW  (Figure  7) 

5.  Horizontal  indicator  bar  with  pop-up  window  (Figure  8) 

6.  Horizontal  indicator  bar  with  AIW  (Figure  8) 

7.  Ticker  and  fading  bar  with  pop-up  window  (Figure  14) 

8.  Ticker  and  fading  bar  with  AIW  (Figure  9) 

All  participants  reviewed  the  design  concepts  in  the  same  order  which  means  that  a  potential  order 
effect  could  not  be  determined.  However,  the  presence  or  absence  of  an  order  effect  was  believed 
to  be  inconsequential  for  this  exploratory  research. 

Part  of  the  evaluation  of  the  designs  was  accomplished  by  a  five  part  questionnaire.  The  first 
section  included  demographic  questions  such  as  name,  rank,  number  of  years  experience,  etc.  The 
second  part  of  the  questionnaire  included  15  questions  related  to  the  usability  and  usefulness  of  a 
non-intrusive  alerting  system  in  general.  SMEs  were  asked  to  select  the  rating  (on  a  5-point  scale) 
they  felt  most  appropriate.  Example  questions  included  “These  alerts  would  enhance  our 
knowledge  of  anomalies”  and  “This  alerting  system  would  be  difficult  to  use.” 

The  third  section  of  the  questionnaire  assessed  participant’s  attitudes  towards  each  non-intrusive 
alert  design.  Again,  SMEs  were  asked  to  select  the  rating  (on  a  7-point  scale)  they  felt  most 
appropriate.  Example  questions  included  “The  number  of  alerts  was  easy  to  comprehend”  and 
“The  priorities  of  the  alerts  were  easy  to  comprehend.” 

The  fourth  part  of  the  questionnaire  assessed  the  level  of  preference  for  each  alert  design  across 
three  dimensions;  overall  effectiveness  in  bringing  alerts  to  the  operator’s  attention,  the  method  in 
which  different  alert  priorities  are  presented,  and  the  degree  to  which  all  of  the  required 
information  about  an  alert  is  presented.  For  each  dimension,  participants  were  asked  to  rank  each 
of  the  four  designs;  with  4 1  ’  indicating  the  most  preferred  and  ‘4’  the  least  preferred. 

The  final  section  of  the  questionnaire  included  open-ended  questions  related  to  each  alert  design. 
For  each  alert  design,  participants  were  asked  if  they  thought  the  design  should  be  implemented 
(Yes=l,  No=2).  They  were  then  asked  to  list  their  likes  and  dislikes  for  each  design.  For  the  final 
question  in  this  section,  participants  were  asked  to  rate  the  intrusiveness  of  the  design  based  on  a  7- 
point  scale  (where  l=Not  at  all  Intrusive,  7=Extremely  Intrusive).  Details  of  the  questionnaires  and 
interview  questions  can  be  found  Annexes  B  and  C. 

6.2.4  Procedure 

Each  SME  participated  individually  in  a  walkthrough  of  each  of  the  designs  using  the  PowerPoint 
presentation  on  a  computer  screen,  and  then  completed  the  usability  questionnaire  and  answered  a 
number  of  follow  up  questions  in  an  interview.  Sessions  lasted  on  average  one  hour. 

Two  members  of  the  research  team  lead  the  walkthroughs,  one  leading  the  process  and  the  other 
taking  notes  and  audio  recording  participant  responses.  Participants  were  first  asked  to  read  a  pre¬ 
experiment  information  sheet  and  then  read  and  sign  a  voluntary  consent  form  (see  Annex  A).  A 
member  of  the  research  team  then  described  the  goals  of  the  project  as  well  as  goals  of  an  alert 
system  in  general.  She  then  presented  a  definition  of  non-intrusive  alerts,  an  overview  of  the  alert 
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designs,  and  a  basic  description  of  the  method  to  be  used  for  the  walkthrough.  After  the 
introduction,  participants  were  asked  if  they  had  any  questions  before  they  were  shown  each  alert 
design  in  detail.  A  member  of  the  research  team  then  walked  participants  through  the  four  alert 
designs,  each  with  the  RMP  pop-up  box  and  then  with  the  AIW.  At  different  points  during  the 
walkthrough,  participants  were  questioned  relating  to  the  topics  such  as  priority  levels,  setting  alert 
trip  points  and  parameters  for  alerts.  Participants  were  allowed  to  ask  questions  and  make 
comments  throughout  the  walkthroughs. 

After  the  walkthroughs,  participants  were  asked  to  complete  the  five  part  questionnaire. 
Participants  were  then  asked  some  general  follow  up  questions  that  were  not  previously  addressed 
during  the  walkthroughs.  If  desired,  participants  were  able  to  revisit  the  different  design  concepts 
during  the  questionnaire  and  general  discussion  phases. 

The  interviewers  used  the  following  points  to  guide  the  discussion,  asking  questions  when 
necessary  (depending  on  what  had  already  been  discussed  during  the  presentation  of  the  design 
concepts): 

•  Operator’s  attendance  at  their  desk 

•  Frequency  of  alerts  (by  priority) 

•  Priority  1  alerts  being  intrusive 

•  Ignoring  alerts 

•  Priority  indications 

•  Alert  colour  scheme  (e.g.,  red,  orange,  yellow,  grey) 

•  Font  (e.g.,  size,  colour) 

•  Terminology 

•  Alert  Information  Window 

•  Intuitive  versus  not  intuitive 

•  Intrusiveness  of  alerts 

•  Auditory/tactile  alerts 

•  Flexibility  in  the  location  of  ticker  and  horizontal  indicator  bar 

•  RMP  centred  on  contact 

•  Information  in  the  RMP  pop  up  box 

•  Ability  to  return  to  primary  task 

•  Shift  change  over  problems 

A  summary  of  individual  participant  responses  are  provided  in  detail  in  Section  6.2.3. 

6.3  Results 

6.3.1  Questionnaire  data 

The  questionnaire  data  are  divided  into  two  sections:  quantitative  and  qualitative.  With  the 
exception  of  the  intrusiveness  ratings,  the  majority  of  questionnaire  responses  were  ratings  on  a  5- 
point  scale.  A  7-point  scale  was  used  for  intrusiveness  ratings  as  the  perception  of  intrusiveness 
was  of  primary  concern  for  this  project  and  so  it  was  thought  that  detecting  finer  distinctions 
between  people  would  be  desirable.  Results  suggest,  however,  that  a  5-point  scale  would  likely 
have  been  suitable.  In  addition  to  rating  scales,  participants  were  also  asked  to  rank  the  designs, 
and  provide  their  likes  and  dislikes  of  each  design  in  an  open  ended  format. 
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6.3.2  Quantitative  data 

6.3.2. 1  General  Non-lntrusive  Design  Questions 

The  Usability  and  Usefulness  Questionnaire  assessed  participants  attitudes  on  the  usefulness  of  a 
general  non-intrusive  alerting  system  (see  Table  16). 


Table  16:  Mean  ratings  for  Usability  and  Usefulness  of  Non-lntrusive  Alerting 

System 


Statement 

Answer9 

Mean 

St.  Dev. 

Range 

(scale  =  1-5) 

These  alerts  would  enhance  our  knowledge  of 
anomalies 

Strongly  agree 

4.6 

.53 

4-5 

This  alerting  system  would  be  used  on  a  daily  basis 

Strongly  agree 

4.9 

.38 

4-5 

Tasks  can  be  performed  in  a  straightforward  manner 
using  this  alerting  system 

Strongly  agree 

4.6 

.79 

3-5 

The  thinking  required  to  use  this  alerting  system 
requires  significant  effort 

Disagree 

1.9 

.38 

1-2 

This  alerting  system  would  be  difficult  to  use 

Strongly 

Disagree/Disagree 

2.0 

1.41 

1-5 

This  alerting  system  will  improve  my  situation 
awareness 

Agree 

4.4 

.53 

4-5 

This  alerting  system  would  make  it  easier  to  identify 
anomalies 

Strongly  agree 

4.7 

.49 

4-5 

1  would  find  this  alert  system  useful 

Disagree 

1.7 

.49 

1-2 

1  would  not  ignore  alerts  while  using  this  technology 

Agree 

4.0 

1.00 

2-5 

The  Alert  Information  Window  (AIW)  was  confusing 

Disagree 

2.1 

.38 

2-3 

It  was  easy  to  learn  how  the  AIW  was  represented 

Strongly  agree 

4.7 

.49 

4-5 

The  AIW  had  all  the  necessary  information 

Disagree 

2.9 

1.07 

2-4 

It  was  easy  navigating  between  the  RMP  and  the  AIW 

Agree 

4.1 

.38 

4-5 

1  prefer  clearing  and  deferring  alerts  directly  from  the 
RMP 

Undecided 

3.1 

1.07 

2-5 

1  prefer  using  the  AIW  to  clear  or  defer  alerts 

Undecided 

3.1 

1.07 

2-5 

As  shown  in  Table  16,  participants  showed  generally  favourable  attitudes  towards  a  non-intrusive 
alerting  system.  Specifically,  participants  strongly  agreed  that  the  non-intrusive  alerting  system 
would  enhance  their  knowledge  of  maritime  anomalies  (Mean  =  4.6),  be  used  on  a  daily  basis 
(Mean  =  4.9),  perform  tasks  in  a  straightforward  manner  (Mean  =  4.6),  and  make  it  easier  to 


9  Scale  descriptor  is  based  on  the  most  frequent  rating  (mode). 
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identify  anomalies  (Mean  =  4.7).  Despite  these  positive  answers,  participants  reported  that  they 
would  not  find  the  alert  system  useful  (Mean  =  1 .7).  This  rating  is  somewhat  surprising  and 
participant  comments  shed  no  light  on  the  reasons  that  may  have  led  to  this  rating.  Therefore,  the 
way  in  which  operators  consider  the  potential  usefulness  of  an  alerting  system  for  maritime 
anomalies  clearly  needs  to  be  addressed  in  future  work.  Participants  were  also  undecided  about 
whether  the  Alert  Information  Window  (AIW)  had  all  the  necessary  information  and  whether  they 
preferred  clearing  alerts  directly  from  the  RMP  or  the  AIW.  Follow  up  interview  questions  covered 
these  issues  and  will  be  discussed  in  Section  6.2.4. 

6. 3. 2. 2  Specific  Design  Questions 

The  rest  of  the  questions  assessed  the  participant’s  attitudes  regarding  each  specific  alerting  design. 
The  following  table  shows  the  means  and  standard  deviations  for  each  question. 


Table  17:  Mean  Ratings  for  the  Alert  Design  Questions 


Statement  &  Design 

Answer10 

Mean 

St.  Dev. 

Range 
(scale  =  1-5) 

The  number  of  alerts  were  easy  to  comprehend 

Design  1:  Cumulative  Total  Indicator 

Strongly  agree 

4.7 

.49 

4-5 

Design  2:  Vertical  Cumulative  Indicator 

Agree 

3.9 

.69 

3-5 

Design  3:  Horizontal  Indicator  Bar 

Agree 

3.9 

1.07 

2-5 

Design  4:  Ticker  &  Fading  Bar 

Agree 

4.1 

.06 

3-5 

The  presence  of  an  alert  was  easy  to  recognize 

Design  1:  Cumulative  Total  Indicator 

Agree 

4.3 

.49 

4-5 

Design  2:  Vertical  Cumulative  Indicator 

Agree 

3.9 

.90 

2-5 

Design  3:  Horizontal  Indicator  Bar 

Agree 

3.7 

1.25 

2-5 

Design  4:  Ticker  &  Fading  Bar 

Strongly  Agree 

4.4 

1.13 

2-5 

The  priorities  of  the  alerts  were  easy  to  comprehend 

Design  1:  Cumulative  Total  Indicator 

Strongly  agree 

4.6 

.53 

4-5 

Design  2:  Vertical  Cumulative  Indicator 

Agree 

4.4 

.53 

4-5 

Design  3:  Horizontal  Indicator  Bar 

Agree 

3.9 

1.07 

2-5 

Design  4:  Ticker  &  Fading  Bar 

Strongly  Agree 

4.4 

.79 

3-5 

It  was  easy  to  find  the  relevant  anomaly 

Design  1:  Cumulative  Total  Indicator 

Agree 

4.1 

.69 

3-5 

Design  2:  Vertical  Cumulative  Indicator 

Agree 

4.0 

.58 

3-5 

Design  3:  Horizontal  Indicator  Bar 

Agree 

4.1 

.69 

3-5 

Design  4:  Ticker  &  Fading  Bar 

Strongly 

Agree/Agree 

4.3 

.76 

3-5 

It  was  easy  to  find  information  on  anomalies 

Design  1:  Cumulative  Total  Indicator 

Agree 

4.3 

.49 

4-5 

10  Scale  descriptor  is  based  on  the  most  frequent  rating  (mode). 
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Statement  &  Design 

Answer10 

Mean 

St.  Dev. 

Range 
(scale  =  1-5) 

Design  2:  Vertical  Cumulative  Indicator 

Agree 

3.9 

.38 

34 

Design  3:  Horizontal  Indicator  Bar 

Agree 

4.0 

.58 

3-5 

Design  4:  Ticker  &  Fading  Bar 

Agree 

4.1 

.69 

3-5 

The  alerting  design  enhanced  my  situation  awareness  of  maritime  anomalies 

Design  1:  Cumulative  Total  Indicator 

Agree 

4.1 

.38 

4-5 

Design  2:  Vertical  Cumulative  Indicator 

Agree 

4.0 

.58 

3-5 

Design  3:  Horizontal  Indicator  Bar 

Agree 

4.0 

.58 

3-5 

Design  4:  Ticker  &  Fading  Bar 

Strongly  agree 

4.6 

.53 

4-5 

The  appearance  of  the  alerts  is  compatible  with  my  current  interface 

Design  1:  Cumulative  Total  Indicator 

Strongly  Agree 

4.0 

1.15 

2-5 

Design  2:  Vertical  Cumulative  Indicator 

Agree 

3.6 

.98 

2-5 

Design  3:  Horizontal  Indicator  Bar 

Undecided/Agree 

3.3 

.76 

2-4 

Design  4:  Ticker  &  Fading  Bar 

Agree 

4.0 

1.00 

2-5 

A  parametric  one-way  Analysis  of  Variance  (ANOVAs)  and  non-parametric  tests  were  conducted 
and  revealed  no  significant  differences  in  ratings  between  the  four  alert  designs  for  any  of  the 
above  questions.  As  a  result,  this  section  presents  only  observations  on  the  trends  in  the  data,  which 
need  to  be  validated  through  further  experimentation. 

Participants  rated  the  Cumulative  Total  Indicator  as  the  easiest  to  comprehend,  while  the  Vertical 
Cumulative  Indicator  and  the  Horizontal  Indicator  Bar  were  equally  more  difficult  to  comprehend. 
The  presence  of  new  alerts  was  easiest  to  recognize  in  the  Ticker  and  Fading  Bar,  followed  by  the 
Cumulative  Total  Indicator,  the  Vertical  Cumulative  Indicator,  and  then  the  Horizontal  Indicator 
Bar.  Alert  priorities  in  the  Cumulative  Total  Indicator  were  easiest  to  comprehend,  and  most 
difficult  in  the  Horizontal  Indicator  Bar.  The  results  also  suggest  that  participants  found  that  the 
Ticker  and  Fading  Bar  was  the  easiest  design  in  which  to  find  the  anomaly,  while  the  Vertical 
Cumulative  Indicator  was  the  most  difficult.  The  Cumulative  Total  Indicator  was  rated  the  easiest 
to  find  the  relevant  information  pertaining  to  the  anomaly,  while  the  Vertical  Cumulative  Indicator 
was  rated  the  hardest  to  find  the  information.  The  Ticker  and  Fading  Bar  enhanced  the  operator’s 
situation  awareness  of  maritime  anomalies  (i.e.,  incoming  alerts  and  total  number  of  active  alerts  in 
the  system),  while  the  Vertical  Cumulative  Indicator  and  the  Horizontal  Indicator  Bar  were  tied  for 
least  likely  to  enhance  situation  awareness.  Lastly,  the  Cumulative  Total  Indicator  and  the  Ticker 
and  Fading  Bar  were  rated  the  most  compatible  with  the  current  interface,  while  the  Horizontal 
Indicator  Bar  was  rated  the  least  compatible. 

Participants  were  also  asked  to  rank  the  designs  in  order  of  preference  (i.e.,  1,  2,  3  and  4)  from  the 
perspective  of  alert  priorities  (Figure  15),  the  degree  to  which  the  design  grabbed  the  attention  of 
the  operator  (Figure  16),  and  providing  relevant  information  without  distraction  (Figure  17).  All 
three  figures  show  a  frequency  count  of  the  specific  rank  for  each  of  the  four  alert  design  concepts. 
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□  Cumulative  Total  Indicator 

■  Vertical  Cumulative  Indicator 

□  Horizontal  Indicator  Bar 

□  Ticker  &  Fading  Bar 


Figure  15:  Subject  rankings  for  effectiveness  in  bringing  alerts  to  my  attention 
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□  Cumulative  Total  Indicator 

■  Vertical  Cumulative  Indicator 

□  Horizontal  Indicator  Bar 

□  Ticker  &  Fading  Bar 
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Priority  Rankings 


Figure  16:  Subject  rankings  for  methods  for  representing  the  priorities 
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□  Cumulative  Total  Indicator 

■  Vertical  Cumulative  Indicator 

□  Horizontal  Indicator  Bar 

□  Ticker  &  Fading  Bar 
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Information  Rankings 


Figure  17:  Subject  rankings  for  providing  required  information  without  distraction 

Kruskal- Wallis  ANOVA  and  Chi-Square  analyses  revealed  no  significant  differences  between  the 
four  alert  designs  in  terms  of  effectiveness  in  bringing  alerts  to  the  operator’s  attention,  methods  of 
representing  priorities  and  providing  required  information  without  distraction.  Though  not 
statistically  significant,  a  trend  analysis  suggests  that  the  Ticker  and  Fading  Bar  was  the  preferred 
design  for  all  of  these  three  features  while  the  Vertical  Cumulative  Indicator  was  the  least 
preferred.  Further  empirical  research  is  needed  to  verify  this  trend. 

6. 3. 2. 3  Implementation  and  intrusiveness 

Participants  were  asked  if  each  design  should  be  implemented  and  they  were  also  asked  to  rate  the 
intrusiveness,  as  shown  in  Table  18. 
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Table  18:  Mean  ratings  for  design  implementation  and  intrusiveness 


Statement  &  Design 

Answer11 

Means 

St.  Dev. 

Range 

Should  the  design  be  implemented 

Design  1:  Cumulative  Total  Indicator 

Yes 

1.3 

.49 

1-2 

Design  2:  Vertical  Cumulative  Indicator 

No 

1.9 

.38 

1-2 

Design  3:  Horizontal  Indicator  Bar 

No 

1.6 

.53 

1-2 

Design  4:  Ticker  &  Fading  Bar 

Yes 

1.0 

.00 

1 

Intrusiveness12  (scale  =  1-7) 

Design  1:  Cumulative  Total  Indicator 

Not  at  all  intrusive 

2.1 

.90 

1-4 

Design  2:  Vertical  Cumulative  Indicator 

Somewhat  intrusive 

2.9 

1.07 

1-4 

Design  3:  Horizontal  Indicator  Bar 

Somewhat  intrusive 

3.7 

1.60 

1-5 

Design  4:  Ticker  &  Fading  Bar 

Somewhat  intrusive 

3.1 

1.46 

1-5 

A  one-way  parametric  ANOVA  and  non-parametric  tests  revealed  no  significant  differences 
between  the  four  alert  designs  in  terms  of  which  design  should  be  implemented  or  the  intrusiveness 
of  each  design.  Though  not  statistically  significant,  all  participants  felt  that  the  Ticker  and  Fading 
Bar  (Mean  =  1.0)  should  be  implemented  and  all  but  two  participants  felt  that  the  Cumulative  Total 
Indicator  (Mean  =  1.3)  should  be  implemented.  On  the  other  hand,  only  one  participant  indicated 
that  the  Vertical  Cumulative  Indicator  (Mean  =1.9)  should  be  implemented  and  three  felt  that  the 
Horizontal  Indicator  Bar  (Mean  =  1.6)  should  be  implemented13. 

Although  not  statistically  significant,  participants  rated  all  four  designs  as  not  at  all  intrusive  or 
somewhat  intrusive.  The  horizontal  indicator  bar  was  rated  the  most  intrusive  and  the  cumulative 
total  indicator  the  least  intrusive.  Interestingly,  as  noted  above,  all  participants  felt  that  the  Ticker 
and  Fading  Bar  should  be  implemented  even  though  it  did  not  receive  the  least  intrusive  rating. 

In  summary,  the  Cumulative  Total  Indicator  and  the  Ticker  and  Fading  Bar  were  rated  more 
favourably  than  the  Vertical  Cumulative  Indicator  and  the  Horizontal  Indicator  Bar.  Specifically,  in 
comparing  the  rank  ordering  of  the  designs,  the  Ticker  and  Fading  Bar  was  preferred  over  all  other 
designs,  while  the  Vertical  Cumulative  Indicator  was  the  least  preferred.  What  is  interesting  to  note 
is  the  Ticker  and  Fading  Bar  includes  the  Vertical  Cumulative  Indicator  scale  with  the  only 
difference  being  priority  colour  in  the  boxes.  Based  on  discussions,  participants  preferred  the  actual 
ticker/fading  bar  that  appeared  at  the  bottom  of  the  screen,  rather  than  the  scale,  which  will  be 
discussed  later. 

6.3.3  Qualitative  data 

The  following  table  shows  the  participants’  qualitative  assessments  of  each  alert  design. 


11  Scale  descriptor  is  based  on  the  most  frequent  rating  (mode). 

12  The  intmsiveness  was  on  a  7-point  scale  (1  =  Not  at  all  intrusive,  4  =  Somewhat  intrusive,  7  =  Extremely 
intrusive) 

13  Yes  was  considered  a  1,  while  No  was  considered  a  2. 
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Table  19:  Likes  and  Dislikes  of  Alert  Designs 


Design 

Likes 

Dislikes 

Comments 

1:  Cumulative 
Total 

Indicator 

Simple  and  clear;  uses  less 
screen;  numeric  total  subtle  but 
effective 

No  hover  feature  for  listing  alerts 
without  leaving  RMP 

This  feature  did  not  exist  for 
any  of  the  design  concepts, 
but  could  be  readily 
implemented. 

Unobtrusive;  not  overbearing  but 
available  as  a  quick  visual 
reference 

This  is  not  good  for  operators  that 
sit  for  hours  in  front  of  this  system 

Comments  made  during  the 
discussion  suggest  that  this 
design  may  be  more  easily 
ignored  by  operators 
especially  over  long  periods 

Exact  priority  of  each  alert; 
unobtrusive  display 

Not  as  immediately  visible  as  the 
ticker/bar 

Compact,  clear  number 

Small  numbers  could  create  a 
change  in  priority  for  an  operator, 
contrary  to  command's  need 

Number  boxes  could  easily 
be  made  larger 

Dead  simple 

No  immediate  visual  of  number  of 
each  type  of  alert  without  focusing 
on  small  numbers.  Also,  location 
should  be  able  to  be  moved  at  will 

The  size  and  location  of  alert 
could  easily  be  changed  in 
future  versions.  Also 
numbers  could  easily  be 
made  larger. 

Does  not  fill  the  RMP  with 
unnecessary  info 

2:  Vertical 

Cumulative 

Indicator 

Visual  and  numerical 
representation  of  alert  number 
and  type;  small  display 

Number  scale  could  be  confused 
with  alert  totals  or  other  data 

Unsure  what  is  meant  by  this 
comment,  especially  “other 
data’’ 

Not  too  intrusive 

Location  should  be  able  to  be 
moved  at  will 

Fairly  simple 

Difficult  to  determine  number  of 
alerts  in  bar 

Not  annoying 

Scale  would  have  to  change  as 
alerts  increase 

Can  become  complacent 

Does  not  give  accurate  info  on 
alerts 

This  is  not  good  for  operators  that 
sit  for  hours  in  front  of  this  system 

Comments  made  during  the 
discussion  suggest  that  this 
design  may  be  more  easily 
ignored  by  operators 
especially  over  long  periods 

3:  Horizontal 
Indicator  Bar 

Easy  to  interpret;  unsure  if  the 
colour  saturation  is  prominent 
enough  when  alerts  increase 

Takes  up  too  much  screen;  scale 
changes  with  number  s  alerts; 
don’t  like  gradual  colour  change 

But  occupies  about  the  same 
amount  of  screen  as  the 
ticker  design. 
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Design 

Likes 

Dislikes 

Comments 

Catches  your  eye  allows  for  quick 
scan  without  refocusing  attention 
from  priority  tasking 

No  number  available  for  alerts; 
colour  saturation  can  be  hard  to 
detect;  takes  up  a  lot  of  bottom 
screen  space 

Clear;  it  would  be  at  the  forefront 

Takes  more  space 

of  the  operator’s  mind 

Too  wide;  won't  notice  overflow  of 
alerts  easily 

Takes  up  too  much  space  on  RMP 
screen 

4:  Ticker  & 
Fading  Bar 

This  would  also  keep  the 
operator  on  the  ball 

Still  has  the  vertical  scale  that 
could  be  confusing 

This  could  easily  be  removed 

Ticker  for  high  priority  alerts  and 
ability  for  operator  to  see 
information  on  cause 

Recommended  fading  grey  bar  in 
and  out  (like  priority  2  ticker)  when 
priority  3  alert  arrives 

But  this  would  produce  some 
potential  confusion  between 
different  alert  priorities.  In 
addition,  priority  3  alerts  are 
not  defined  as  events  that 
should  immediately  capture 
attention. 

Decreasing  intrusiveness;  wide 
area  only  used  for  incoming 
alerts 

Don't  like  ticker;  fading  bar  should 
be  option;  some  operators  will  like 
and  some  won't 

Allows  immediate  connection 
with  track  in  RMP;  less  steps  to 
view 

Fade  bar  quite  distracting;  no  real 
info  contained 

However,  this  is  no  different 
from  having  a  flashing 
indicator,  as  in  the  cumulative 
counter  design. 

With  information  in  the  bar,  the 
operator  does  not  have  to 
change  what  they  are  doing 

This  is  not  really  the  case,  as 
the  operator  will  have  to  shift 
attention  momentarily  to 
process  the  information  in  the 
bar. 

Most  prominent;  easy  to 
distinguish  levels 

Prominence  may  become  a 
negative  factor  in  a  high 
frequency  alert  context. 

As  shown  in  Table  19,  some  of  the  dislikes  were  implementation  issues  and  the  designs  could  be 
easily  changed  or  modified  to  accommodate  the  suggestions.  There  was  a  comment  for  the  Vertical 
Cumulative  Indicator  that  needs  further  explanation.  The  participant  who  said,  “This  is  not  good 
for  operators  that  sit  for  hours  in  front  of  this  system”  and  noted  “. ...  the  design  is  not  stimulating 
enough  for  the  mind”.  The  participant  believed  that  the  operator  needs  to  be  “stimulated”  and  their 
mind  “needs  to  be  active”,  and  believed  that  with  Design  2,  this  would  not  happen. 

Overall,  the  Cumulative  Total  Indicator  was  evaluated  as  being  non-intrusive  and  very  simple  to 
use  and  interpret.  However,  some  participants  felt  that  it  was  too  small  and  could  therefore  be 
easily  ignored  by  operators.  It  appears  that  there  needs  to  be  a  compromise  relating  to  the  size  of 
the  alert  such  that  it  is  large  and  salient  enough  to  be  noticed  yet  not  too  salient  as  to  distract  the 
operator  from  his/her  primary  task  or  become  an  annoyance. 
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The  Vertical  Cumulative  Indicator  was  also  rated  as  non-intrusive  yet  participants  found  the  scale 
more  confusing  to  interpret  than  the  digital  indicator  in  the  Cumulative  Total  Indicator. 

Only  three  participants  liked  some  aspects  of  the  Horizontal  Indicator  Bar,  while  most  participants 
felt  the  display  obscured  too  much  of  the  screen.  Although  the  Ticker  and  Fading  Bar  used  up  as 
much,  if  not  more  space  than  the  Horizontal  Indicator  Bar,  participants  did  not  mention  this  issue 
for  the  former.  It  is  clear  that  such  inconsistencies  may  have  biased  or  influenced  the  results. 
Hence,  further  research  is  required  to  explain  these  inconsistencies. 

There  were  two  components  to  the  Ticker  and  Fading  Bar  design  that  did  not  exist  with  the  other 
designs.  First,  the  ticker  and  fading  bar  itself  alerted  operators  to  individual  incoming  alerts. 
Second,  a  counter  similar  to  that  of  the  Vertical  Cumulative  Indicator  was  used  to  depict  the  total 
number  of  active  alerts  in  the  system  (i.e.,  alerts  that  hadn’t  been  addressed).  It  is  not  surprising 
then  that  participants  generally  did  not  like  the  scale  (which  is  consistent  with  the  comments  on  the 
Vertical  Cumulative  Indicator).  A  number  of  participants  recommended  that  the  ticker  or  fading 
bar  be  used  for  both  priority  1  and  2  alerts  (i.e.,  there  was  no  need  to  have  the  ticker  for  only 
priority  1  alerts  and  the  fading  bar  for  priority  2  alerts).  Most  participants  preferred  the  Ticker  and 
Fading  Bar  because  of  the  intrusiveness  (i.e.,  it  was  easily  noticed)  and  the  information  contained 
in  the  display  bar.  This  result  may  be  different,  of  course,  if  there  are  frequent  alerts.  Further 
research  in  an  environment  with  a  representative  number  of  alerts  should  be  conducted  to  validate 
these  findings. 

After  participants  completed  the  questionnaires,  they  were  asked  some  additional  follow-up 
questions.  These  questions  related  generally  to  alerting  parameters,  information  presentation  in  the 
RMP  and  potential  challenges  of  an  alerting  system  in  general  and  of  our  visual  designs. 

6.3.4  General  discussion  with  participants 

The  following  points  were  discussed  with  participants  following  the  design  walkthrough  and 
administration  of  the  questionnaire. 

6.3.4. 1  Alert  parameters 

For  the  purpose  of  this  study,  we  assumed  that  the  alert  parameters  operators  would  want  to  set 
included  area  (co-ordinates),  vessel  type  (e.g.,  fishing,  warship,  merchant,  etc),  speed,  and  alert 
priority  (1,  2  or  3).  Participants  stated  that  they  would  also  like  to  set  alert  parameters  according  to 
the  activity  of  the  vessel,  vessel  name,  threat  level  (e.g.,  friendly,  neutral,  suspect),  age  of  alert, 
type  of  anomaly  (e.g.,  not  heading  to  port),  flag  (i.e.,  country),  course  direction,  estimated  time  of 
arrival,  AIS  number14,  and  next  port  of  call.  They  emphasized  that  changing  alert  parameters 
should  be  done  quickly  and  easily,  and  they  should  have  the  ability  to  change  these  parameters 
geographically  depending  on  their  area  of  interest  at  a  given  time.  The  example  given  was  that  if 
you  were  monitoring  Europe’s  coast,  operators  would  not  want  alerts  relating  to  all  vessels  in  that 
area.  Thus,  the  operators  require  the  ability  to  quickly  and  easily  set  these  alerts  according  to  their 
current  area  of  interest. 


14  AIS  number  is  used  to  identify  ship  transponder  to  radio  receiving  station  while  MMSI  number  does  the 
same  for  satellite  receiving  stations 
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6. 3.4. 2  Intrusiveness  and  priority 

Participants  agreed  that  the  level  of  intrusiveness  of  an  alert  must  vary  according  to  the  priority  of 
the  alert.  For  the  purpose  of  this  study,  we  assumed  three  priority  levels  would  be  appropriate  and 
for  the  most  part,  participants  agreed.  A  few  participants  suggested  having  4  or  2  alert  priorities 
instead  of  3.  Furthermore,  several  participants  added  the  caveat  that  priority  3  alerts  will  be  ignored 
90%  of  the  time. 

6.3 A. 3  Inability  to  see  visual  alerts 

When  asked  how  often  operators  could  be  away  from  their  desks  and  the  RMP,  in  turn,  unable  to 
be  notified  of  alerts,  participants  reported  that  there  are  manning  issues  on  a  regular  basis  and 
therefore  operators  are  needed  elsewhere.  Despite  this  challenge,  participants  did  not  like  the  idea 
of  an  auditory  or  tactile  alert  (e.g.,  vibrating  pager)  because,  although  the  operator  would  be 
notified  of  the  alert,  he  or  she  would  not  be  able  to  get  back  to  their  workstation  to  deal  with  the 
alert.  Furthermore,  most  participants  felt  that  operators  would  elect  not  to  use  such  a  pager-type 
device.  Rather,  they  felt  that  the  visual  alert  should  continue  to  be  displayed  on  the  RMP  until  the 
operator  acknowledges  it  (as  is  the  case  with  the  design  concepts  presented) 

6.3.4A  Workload 

In  terms  of  workload,  participants  admitted  that  this  type  of  alert  design  would  increase  the 
operator’s  workload;  however,  all  believed  the  anomaly  alerting  system  would  be  worthwhile 
because  currently  most  such  anomalies  are  missed. 

6.3 A.  5  Colour  coding 

All  the  participants  liked  the  colour  scheme  that  was  used  to  depict  priority  levels  in  the  alert 
designs  (i.e.,  red  for  priority  1,  gold  for  priority  2  and  grey  for  priority  3).  In  the  current  RMP,  red, 
yellow  and  green  are  used,  but  participants  felt  that  green  would  be  inappropriate  for  priority  3 
alerts.  As  one  participant  stated,  “Green  doesn’t  communicate  the  right  message.  Green  means  go.” 

6. 3.4. 6  Readability 

Participants  felt  that  the  size  of  the  displays  and  fonts  were  generally  acceptable,  with  the  exception 
of  two  participants  who  felt  that  the  numbers  in  the  Cumulative  Indicator  were  too  small. 
Furthermore,  one  participant  suggested  that  the  font  in  the  pop-up  windows  should  be  slightly 
larger. 

6.3A.7  Location  on  the  screen 

All  participants  also  liked  the  idea  of  having  the  ability  to  click  and  drag  the  anomaly  alert  display 
anywhere  on  the  screen.  The  alert  designs  presented  to  the  participants  in  the  trial  had  the  alert 
display  either  in  the  bottom  right  hand  comer  or  covering  the  entire  width  of  the  bottom  of  the 
screen.  Participants  thought  this  may  impair  their  situation  awareness  if  they  had  to  focus  on  the 
bottom  of  the  RMP  for  any  length  of  time.  Similarly,  the  current  RMP  has  the  ability  to  centre 
around  a  vessel  of  interest  or  area  of  interest.  Participants  indicated  that  they  would  like  this  to  be 
an  option  for  the  non-intmsive  design  as  well.  That  is,  they  would  like  the  ability  to  centre  the 
picture  on  a  contact  for  which  there  is  anomalous  information.  This  would  allow  an  operator  to 
understand  the  immediate  area  and  then  the  surroundings  around  the  contact  of  interest.  One 
participant  mentioned  that  the  option  to  centre  the  RMP  on  the  contact  of  interest  could  be 
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implemented  as  a  button  in  the  AIW.  For  example,  clicking  on  the  name  of  the  ship  could  centre 
the  RMP  on  the  vessel  of  interest. 

6.3 A. 8  Retrieving  information  on  alert 

As  previously  described,  participants  were  shown  two  options  to  retrieve  information  about  an 
alert.  In  one  case,  the  RMP  had  a  pop  up  box  coming  from  the  vessel  of  interest  which  included  the 
name  of  the  vessel,  reason  for  alert  (e.g.,  grab  and  dash  fishing),  the  time  of  the  alert,  and  action 
buttons  (i.e.,  Clear  or  Leave).  While  most  of  the  participants  thought  the  RMP  pop  up  box  included 
enough  information,  one  participant  recommended  that  it  include  course,  speed,  latitude,  longitude, 
Maritime  Mobile  Service  Identity  (MMSI)  number  (or  the  AIS  #).  The  second  way  to  get 
information  about  an  alert  was  to  go  to  the  AIW  which  was  on  a  separate  screen.  As  designed,  the 
AIW  also  functions  as  a  tool  to  manage  all  alerts  as  it  shows  all  the  alerts  in  the  system  according 
to  priority  level,  total  number  of  alerts  by  priority,  track  number,  name  of  vessel,  reason  for  alert, 
time  of  alert,  and  action  buttons  (i.e.,  Clear,  Done  and  RMP,  which  is  a  button  that  takes  you  back 
to  the  RMP).  Participants  recommended  that  the  AIW  also  include  who  reported  the  alert  (i.e., 
source),  vessel  type,  flag,  age  of  alert  (instead  of  time  of  alert),  time  of  last  report  (as  well  as  the 
ability  to  go  to  that  report),  age  of  track,  course,  speed,  and  the  history  of  the  alert  (e.g.,  if  the 
vessel  has  had  other  alerts  before  such  as  speeding  up  when  it  should  not).  Participants  mentioned 
that  track  number  is  not  useful  and  should  be  exchanged  for  AIS  number.  It  was  also  recommended 
that  a  smaller  version  of  the  AIW  pop  up  in  the  RMP  (similar  to  the  pop-up  box)  rather  than  taking 
the  operator  to  a  separate  screen  as  navigating  away  from  the  RMP  can  adversely  impact  their 
situation  awareness. 

Additionally,  the  alert  designs  included  a  flashing  circle  around  the  contact  of  interest,  but  only  in 
the  designs  using  the  AIW  (the  pop-up  window  points  to  the  contact  of  interest  so  there  is  no  need 
for  a  circle).  The  circle  was  red  for  priority  1  alerts,  gold  for  priority  2  alerts  and  grey  for  priority  3 
alerts.  Two  options  for  this  design  were  shown  to  participants.  First,  the  circle  appeared  around  the 
contact  of  interest  as  soon  as  the  operator  clicked  on  an  incoming  alert  (i.e.,  before  going  to  the 
AIW).  The  other  option  was  to  navigate  to  the  AIW  first  then  click  on  the  RMP  button  to  come 
back  to  the  RMP  on  which  the  flashing  circle  would  appear  around  the  contact.  All  participants 
preferred  when  this  circle  appeared  as  soon  as  the  alert  was  clicked  (i.e.,  before  going  to  the  AIW). 
However,  participants  indicated  that  they  generally  preferred  the  pop-up  window  over  the  AIW  as 
the  source  for  anomaly  information  which  means  that  the  flashing  circle  would  not  be  required. 

In  summary,  for  the  seven  participants  there  were  eleven  overall  favourite  designs,  due  to 
participants  preferring  two  of  the  designs  equally.  Five  of  the  seven  participants  preferred  the 
Ticker  and  Fading  Bar,  four  participants  preferred  the  Cumulative  Total  Indicator,  and  one 
participant  preferred  the  Horizontal  Indicator  Bar.  There  was  only  one  participant  who  favoured  the 
Vertical  Cumulative  Indicator;  however,  with  the  caveat  that  it  had  to  be  in  conjunction  with  the 
Ticker  and  Fading  Bar. 

Thus,  overall  the  Ticker  and  Fading  Bar  was  favoured  because  it  was  the  most  prominent  and 
participants  liked  the  information  provided  in  the  ticker.  The  participants  who  favoured  the 
Cumulative  Total  Indicator  did  so  because  it  was  the  least  intrusive  and  they  could  see  the  exact 
number  of  alerts  in  the  system.  The  Vertical  Cumulative  Indicator  was  disliked  because  of  the  scale 
and  the  fact  that  it  was  difficult  to  see  the  exact  number  of  alerts  in  the  system.  Further,  they  were 
concerned  about  the  amount  of  space  it  would  take  up  on  the  screen  if  there  were  a  lot  of  alerts  and 
the  scale  needed  to  be  increased  (a  0-10  scale  was  used  for  the  design).  The  Horizontal  Indicator 
Bar  was  disliked  because  it  was  difficult  to  determine  the  actual  number  of  alerts  and  the  colour 
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saturation  would  be  difficult  to  learn.  However,  one  participant  believed  that  this  design  requires 
some  cognitive  effort  to  interpret  which  could  be  positive  in  that  it  would  “keep  the  operator’s 
mind  busy”  (i.e.,  maintaining  or  improving  attention  and  vigilance).  On  the  other  hand,  this 
participant  felt  that  the  Cumulative  Total  Indicator  and  the  Vertical  Cumulative  Indicator  could  be 
easily  ignored  as  they  do  not  require  as  much  mental  effort  to  interpret.  These  comments  suggest 
that  the  appropriate  level  of  intrusiveness  for  a  maritime  alert  system  for  RJOC  operators  is  not 
established  and  must  be  further  researched  and  determined  experimentally,  particularly  under 
operational  contexts  of  medium  to  high  alert  frequency. 

6.4  Conclusions  and  recommendations  based  on  evaluation 

This  section  presents  conclusions  and  a  number  of  recommendations  for  future  anomaly  alert 
system  designs  based  on  the  SME  review  and  evaluation. 

6.4.1  Conclusions 

In  terms  of  presentation  of  incoming  alerts ,  participants  favoured  both  the  Ticker  and  Fading  Bar, 
which  was  rated  as  fairly  intrusive  and  the  Cumulative  Total  Indicator,  which  was  rated  as 
relatively  non-intrusive.  Further  research  is  therefore  required  to  determine  the  appropriate  level  of 
intrusiveness  especially  once  the  potential  number  alerts  that  may  be  present  in  the  system  is  better 
understood. 

In  terms  of  presentation  of  the  number  of  active  alerts  in  a  system,  participants  favoured  a 
numerical  display  rather  than  a  scale  as  it  was  easier  to  interpret  at  a  quick  glance. 

With  regards  to  getting  information  about  an  incoming  alert,  participants  preferred  the  pop  up 
window  in  the  RMP  compared  to  the  AIW.  This  was  primarily  because  the  way  in  which  the  AIW 
was  implemented  in  the  prototype;  it  required  the  user  to  navigate  to  a  separate  window. 

Participants  felt  that  this  adversely  impacted  their  situation  awareness  as  a  result.  This  suggests  that 
managing  alerts  should  be  implemented  in  such  a  way  that  operators  do  not  have  to  leave  the 
RMP.  For  example,  a  separate  window  or  sidebar  could  pop  up  in  the  RMP  thereby  allowing  the 
operators  to  still  have  the  RMP  in  view  while  managing  alerts. 

Finally,  participants  felt  that  the  AIW  would  be  appropriate  for  managing  alerts,  although  they 
suggested  a  number  of  additional  pieces  of  information  that  it  should  include. 

Although  the  SME  feedback  from  the  design  review  was  valuable,  the  results  must  be  interpreted 
with  caution  given  the  small  sample  size  and  inconsistency  in  the  comments.  Further  research  is 
required  to  understand  these  inconsistencies  that  may  have  biased  or  influenced  the  results. 

Although  beyond  the  scope  of  the  current  project,  it  should  be  noted  that  an  anomaly  alerting 
system  will  undoubtedly  require  revision  of  the  current  Standard  Operating  Procedures  (SOPs).  For 
example,  information  relating  to  anomalies  will  have  to  be  passed  on  to  incoming  watch  keepers 
during  watch  handover.  That  is,  operators  will  at  least  be  required  to  pass  on  the  current  alert 
parameter  preferences  (i.e.,  the  alert  trip  points)  as  well  as  a  summary  of  the  number  and  types  of 
alerts  that  have  emerged  over  the  course  of  the  shift.  Also,  the  operator  may  be  required  to  brief  the 
Watch  Officer  on  any  Priority  1  alerts  immediately  and  perhaps  keep  a  log  of  all  priority  2  and  3 
alerts.  Furthermore,  which  alerts  have  been  briefed  to  the  Watch  Officer  and  the  time  of  the 
briefing  may  be  an  additional  piece  of  information  that  should  be  included  in  the  AIW. 
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6.4.2  Limitations 

There  are  three  primary  issues  concerning  the  validity  of  the  data  obtained  from  the  study. 

(i)  Reliability:  the  small  sample  size  means  that  the  data  obtained  should  be  treated  with 
caution  and  should  be  used  to  indicate  trends  in  attitudes  towards  a  non-intrusive  alert 
system.  A  more  extensive  evaluation  should  be  conducted  in  the  future  to  verify  these 
trends  (particularly  as  they  pertain  to  specific  design  elements).  This  evaluation  should 
also  involve  operators  from  the  West  Coast. 

(ii)  Context  validity:  the  evaluation  was  limited  in  scope  by  showing  designs  as  single 
event  representations.  However,  in  considering  a  future  operational  environment,  there 
could  be  potentially  a  continuing  volume  of  alerts  that  occur  during  a  watch. 

Therefore,  the  validity  of  the  judgments  obtained  in  the  walkthrough  concerning  the 
appropriate  level  of  intrusiveness  of  the  design  alternatives  must  be  treated  with 
caution.  A  design  option  that  appears  to  offer  the  right  level  of  intrusiveness  in  alerting 
the  operator,  when  viewed  in  isolation,  may,  over  time  and  with  a  high  frequency  of 
alerts,  become  too  distracting. 

(Hi)  Halo  effects:  it  is  possible  that  if  a  trial  participant  favoured  a  particular  design,  then  all 
detailed  evaluation  questions  concerning  the  design  could  be  tainted  by  this  bias.  For 
example,  if  the  design  were  seen  as  being  insufficiently  intrusive  to  “keep  operators  on 
their  toes”,  then  its  usefulness  and  utility  may  also  have  been  judged  lower. 

6.4.3  Design  recommendations  for  anomaly  alert  system 

Considering  the  participant  feedback,  the  results  suggest  that  an  anomaly  alert  design  which 
combines  the  counter  from  the  Cumulative  Total  Indicator,  to  indicate  the  number  of  active  alerts 
in  the  system,  with  the  Ticker  and  Fading  Bar,  to  notify  the  operator  of  incoming  alerts  (with  an 
option  for  either  one),  would  be  the  next  logical  iteration  of  a  non-intrusive  anomaly  alerting 
system  design. 

While  participant  feedback  from  the  design  walkthroughs  can  be  used  to  point  the  way  toward 
future  iterations  of  an  alert  system  design,  caution  must  be  used  in  interpreting  this  feedback 
especially  given  the  small  sample  size  and  inconsistency  in  the  comments.  Hence,  design 
guidelines  must  also  be  based  on  human  factors  principles  and  further  experimentation  in  a  realistic 
context  (i.e.,  in  the  RJOC  using  the  RMP).  Table  20  shows  a  number  of  recommendations  for 
future  alert  system  designs  based  on  participant  feedback  as  well  as  human  factors  principles. 


Table  20:  Design  recommendations  for  future  iterations  of  alert  system  interface 


Design  feature 

Recommendations  based  on  participant 
feedback 

Recommendations  considering  HF 
principles 

Alert 

Alert  indicator  should  be  visual 

Alert  indicator  should  be  visual 

Alert  indicator  should  only  disappear  after 
acknowledgement 

Alert  indicator  should  only  disappear  after 
acknowledgement 

Ability  to  click  and  drag  the  alert  display  to 
a  different  location  on  the  RMP 

Ability  to  click  and  drag  the  alert  display  to  a 
different  location  on  the  RMP  so  as  not  to 
obscure  the  RMP  (i.e.  primary  task) 

Font  in  the  pop-up  boxes  should  be  a  bit 

Font  in  the  pop-up  boxes  should  be 
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Design  feature 

Recommendations  based  on  participant 
feedback 

Recommendations  considering  HF 
principles 

bigger 

appropriate  for  distance  that  operator  is  sitting 
from  screen  and  size  of  fonts  used  in  primary 
task  and  ambient  illumination 

Time  should  be  in  Zulu  time 

Time  should  be  consistent  with  that  used  for 
current  tasks 

Ticker  and  fading  bar  should  also  include 
the  vessel  name 

The  required  level  of  information  content  to  be 
contained  with  in  alert  indicator  should  be 
further  investigated 

Option  to  centre  the  screen  around  a 
vessel  of  interest  or  area  of  interest 

Option  to  centre  the  screen  around  a  vessel  of 
interest  or  area  of  interest  to  support  SA 

Priority 

•  Priority  1  alerts  will  be  highlighted 
in  red 

•  Priority  2  alerts  will  be  highlighted 
gold 

•  Priority  3  alerts  will  be  highlighted 
in  grey 

Priority 

•  Priority  1  alerts  will  be  highlighted  in 
red. 

•  Priority  2  alerts  will  be  highlighted 
gold. 

•  Priority  3  alerts  will  be  highlighted  in 
grey. 

Incoming  vs.  Active  Alerts 

•  Incoming  alerts  should  be 
indicated  in  a  ticker  or  fading  bar 
(priority  1  and  2)  and/  or  as  a 
cumulative  number  in  the  count 
box  (priority  1 , 2  and  3) 

•  Active  alerts  left  in  the  system 
will  only  be  indicated  as  a 
cumulative  number  in  the  count 
box  (priority  1, 2  and  3) 

Incoming  vs.  Active  Alerts 

•  Presentation  of  incoming  alerts 
should  be  further  investigated  in  a 
context  that  is  representative  of 

RJOC  operator’s  actual  work 
environment. 

•  Active  alerts  left  in  the  system  will  be 
indicated  as  a  cumulative  number  in 
the  count  box  (priority  1 , 2  and  3) 

Retrieving  Information 

The  RMP  pop  up  box,  rather  than  the  AIW, 
should  be  used  to  retrieve  information 
related  to  an  anomaly 

This  solution  represents  a  trade-off  between 
the  amount  of  space  required  for  the 
appropriate  information  content  concerning  the 
anomaly  (which  has  a  potential  for  obscuring 
the  RMP)  and  obtaining  the  information  from  a 
separate  window,  taking  the  operator's 
attention  away  from  the  RMP.  The  appropriate 
design  option  will  require  further  investigation 
before  this  recommendation  can  be  stated 
definitively. 

The  RMP  pop  up  box  will  show  AIS 
number,  name  of  the  vessel,  reason  for 
alert,  age  of  alert,  and  action  buttons 

The  anomaly  information  content  of  the  RMP 
pop  up  box  should  be  further  investigated 

Managing  Alerts 

The  AIW  should  be  used  to  manage  alerts 

The  AIW  should  be  used  to  manage  alerts 

The  AIW  will  show  priority  level,  a  button  to 
return  to  the  RMP,  number  of  alerts,  age  of 
alerts,  AIS  number,  name  of  vessel,  reason 
for  alert,  action  buttons,  source  of  alert, 

The  content  and  functionality  of  the  AIW 
should  be  further  investigated  by  determining 
the  specific  operational  requirements. 
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Design  feature 

Recommendations  based  on  participant 
feedback 

Recommendations  considering  HF 
principles 

vessel  type,  flag,  age  of  alert,  course, 
speed,  alert  history  and  age  of  track 

The  operator  should  not  have  to  navigate 
to  a  different  screen  to  see  the  AIW 

This  cannot  be  supported  without  further 
investigation.  For  example,  by  the  end  of  the 
watch,  and  during  a  high  frequency  alerting 
context,  there  may  be  numerous  alerts  in  the 
system.  These  would  either  have  to  be 
represented  in  a  small  window  on  the  RMP, 
though  which  the  operator  would  have  to 
continually  screen  (i.e.,  not  functionally 
efficient,  or  usable)  or  a  large  window,  which 
would  then  obscure  a  significant  portion  of  the 
RMP. 

The  AIW  may  appear  as  a  pop  up  window 
over  the  RMP  that  can  be  increased  or 
decreased  in  size  as  desired 

The  AIW  may  appear  as  a  pop  up  window  over 
the  RMP  that  can  be  appropriately  sized  for 
the  information  content  up  to  a  certain 
maximum. 

Acknowledging  Alerts 

Acknowledgment  of  an  alert  will  come  in 
the  form  of  LEAVE  or  CLEAR 

The  most  appropriate  terms  for  acknowledging 
an  alert  should  be  intuitive  to  all  users  and  be 
consistent  with  similar  functions  in  the  current 
system,  and  should  therefore  be  further 
investigated 

Ability  to  action  alerts  through  the  RMP  pop 
up  box 

Ability  to  action  alerts  through  the  RMP  pop  up 
box 

Ability  to  action  alerts  through  the  AIW 

Ability  to  action  alerts  through  the  AIW 
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7.  Overall  conclusions  and 
recommendations 


This  section  presents  a  number  of  overall  conclusions  based  on  the  literature  review,  design 
development  process  and  SME  review  and  evaluation  of  the  alert  system  design  concepts. 

7.1  Conclusions 

Our  general  assessment  of  the  literature  relating  to  non-intrusive  alert  system  design  is  that  it  is  a 
somewhat  piecemeal  and  haphazard  collection  of  principles  and  guidelines  from  various 
application  environments  and  serving  a  number  of  different  goals.  Clearly,  there  is  a  lack  of  a 
unified  design  approach  and  associated  recommendations  that  would  be  applicable  for  non- 
intrusive  alerting  contexts.  However,  the  detailed  guidelines  found  in  Shorrock  et  al  (2002)  and 
Han  et  al  (2007)  provide  a  good  starting  point  for  an  integrated  guidance  document,  which  would 
then  need  to  be  extended  and  made  more  context  relevant  to  the  RJOCs.  In  addition,  there  would 
be  a  need  to  make  this  guidance  more  specific  than  statements  such  as  “alarms  should  signal  the 
need  for  action”,  “alarms  should  be  detected  rapidly. . “alarms  should  not  annoy,  startle  or 
distract  unnecessarily”  etc.  Thus,  while  these  recommendations  are  sound,  there  is  a  lack  of 
information  on  how  they  are  to  be  implemented  as  specific  design  guidelines. 

The  review  of  the  literature  gave  rise  to  some  general  principles  for  four  alert  system  design 
concepts  that  were  the  presented  to  and  evaluated  by  SMEs.  The  actual  translation  of  these 
principles  into  specific  designs  was  very  much  based  on  Humanyys' terns'  prior  experience  with  HF 
design  implementation,  rather  than  specific  recommendations  from  the  literature  reviewed.  The 
design  evaluation,  combined  with  consideration  of  general  human  factors  principles,  resulted  in  a 
list  of  design  requirements  for  the  best  way  to: 

•  Alert  RMP  operator  to  a  new  incoming  alert 

•  Provide  operator  with  awareness  of  the  number  of  active  alerts  in  the  system 

•  Provide  operator  with  information  specific  to  an  incoming  alert 

•  Provide  operator  with  information  on  all  active  alerts  in  the  system 

•  Provide  operator  with  a  means  to  acknowledge  the  occurrence  of  an  alert 

•  Enable  operator  to  manage  (i.e.,  action)  any  active  alerts  in  the  system 

Further  research,  however,  is  needed  to  better  clarify  design  options  that  would  support  these 
design  requirements,  particularly  under  more  realistic  operational  conditions  of  multiple  alerts 
within  a  watch. 

7.2  Future  Work 

The  literature  review  and  SME  feedback  from  the  design  review  were  valuable  in  providing  a 
direction  for  both  future  iterations  of  an  anomaly  alert  system  design  as  well  as  future  research. 
Specifically,  future  design  efforts  should  work  toward  developing  an  alert  system  interface  design 
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in  accordance  with  the  design  principles  listed  in  section  6.3.3  once  these  design  requirements  have 
been  validated  through  further  research. 

Future  research  efforts  should  focus  on  both  experimentally  evaluating  anomaly  alert  system 
designs  in  the  context  of  the  RMP  (i.e.,  representative  of  user’s  work  environment  including  the 
potential  number  of  alerts)  as  well  as  broader  research  relating  to  intrusiveness  and  attention.  The 
following  list  provides  a  number  of  research  questions  that  have  yet  to  be  resolved: 

•  What  is  the  appropriate  number  of  alert  priorities? 

•  What  is  the  most  appropriate  level  of  intrusiveness  for  different  priorities  of  alerts  and  how 
is  this  influenced  by  alert  frequency? 

•  What  is  the  relative  intrusiveness  of  a  range  of  design  parameters  such  as  alert  size, 
location  and  dynamics? 

•  How  are  attention  and  intrusiveness  related? 

•  In  an  RMP  context,  what  are  the  trade-offs  between  implementing  the  information  within 
the  RMP  window  and/or  providing  a  separate  window? 

•  In  the  context  of  the  RJOC,  what  are  the  information  requirements  for  an  alert  management 
system  for  data  anomalies? 

•  How  should  the  anomaly  alert  system  be  integrated  within  the  overall  “alert”  system 
currently  used  in  GCCS-M? 

•  Given  the  characteristics  of  the  GCCS-M,  what  are  the  most  appropriate  design 
characteristics  (e.g.,  colour,  font  size,  etc.)  for  an  anomaly  alert  system? 
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Annex  A:  Ethics  Protocol 


EXECUTIVE  SUMMARY 
Protocol  #  L676 

Title:  Evaluation  of  Interface  Designs  for  Non-Intrusive  Alerts:  Pilot  Study 

Principal  Investigators:  Dr.  Michael  Matthews,  Lora  Bruyn  Martin,  stems® 

Incorporated  (HSI®),  Guelph,  Ontario.  Tel:  519-836-5911 

Defence  Research  and  Development  Canada  (DRDC)  Co-Investigators: 

Ms.  Sharon  McFadden,  DRDC  Toronto,  Tel  (416)  635-2189 
Ms.  Liesa  Lapinski,  DRDC  Atlantic.  Tel:  (902)  426-3100  xl80 
Thrust:  1  lhe  Maritime  Domain  Awareness 

Objectives: 

The  goal  of  this  experiment  is  to  explore,  in  the  context  of  the  Recognized  Maritime  Picture 
(RMP),  non-intrusive  ways  of  presenting  to  the  operator  information  concerning  anomalous 
behavior  of  vessels.  Anomalous  behavior  can  take  many  forms,  e.g.  a  sudden  increase  in  speed,  a 
vessel  in  transit  that  suddenly  stops,  a  ship  that  changes  its  port  of  destination  or  a  ship  heading  into 
regulated  waters  without  appropriate  permissions.  Because  of  the  large  number  of  vessels  and 
associated  track  data,  it  is  not  currently  possible  for  operators  to  readily  determine  when,  or  where, 
such  anomalies  occur. 

Overview: 

Six  volunteer  participants  (no  age  or  gender  restrictions)  will  be  required  to  review  a  number  of 
different  design  options  for  a  non-intrusive,  anomaly  alerting  system.  The  participants  will 
individually  do  a  walk  through  of  the  options  which  will  be  presented  as  a  PowerPoint  mock  up  of 
the  screen  interface.  A  questionnaire  will  be  administered  at  the  end  of  each  session,  to  record  the 
volunteers’  subjective  evaluation  of  the  different  design  options  in  terms  of  their  utility  and 
usability.  The  experiment  will  be  undertaken  at  the  MAPLE  laboratory  DRDC  and  will  require 
one  walkthrough  session.  Each  session  will  comprise  approximately  15  minutes  of  background 
briefing  and  a  60-90  minute  walkthrough  session,  which  will  involve  interface  walkthroughs  and 
the  completion  of  a  questionnaire.  This  work  will  be  conducted  by  Humansystems®  Incorporated. 

Participants: 

Male  or  female  Navy  or  ex-Navy  operators  will  be  recruited  and  paid  for  their  participation.  There 
is  no  restriction  on  age  or  gender.  The  two  ex-Navy  operators  are  both  males. 

Risks: 

This  experiment  offers  minimal  risk  to  the  participant’s  health  and  well-being.  There  is  a  low  risk 
of  eye  fatigue  or  eyestrain,  as  is  associated  with  doing  any  visually  intensive  task  on  a  computer 
display. 
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Principal  Investigators: 

Dr.  Michael  Matthews,  Lora  Bruyn  Martin,  Humansystems  Incorporated®  (HSI®),  Guelph, 

Ontario.  Tel:  519-836-5911 

DRDC  Co-Investigators: 

Ms.  Sharon  McFadden,  DRDC  Toronto,  Tel  (416)  635-2189 
Ms.  Liesa  Lapinski,  DRDC  Atlantic,  Tel:  (902)  426-3100 
Thrust:  1  lhe  Maritime  Domain  Awareness 

List  of  Acronyms: 

DRDC  Defence  Research  and  Development  Canada 

HSI®  Humansy stems  Incorporated 

MDA  Maritime  Domain  Awareness 

RMP  Recognized  Maritime  Picture 

RJOCs  Regional  Joint  Operations  Centres 

Background: 

This  applied  research  project  in  the  Maritime  Domain  Awareness  (MDA)  Thrust  is  studying 
information  visualization  and  management  for  enhanced  domain  awareness  in  maritime  security. 
The  DRDC/HSI®  team  wants  to  investigate  the  best  way  to  provide  operators  with  non-intrusive 
alerts  when  certain  forms  of  anomalous  vessel  behavior  occur  to  see  if  the  information  provided  by 
way  of  the  alert  can  help  improve  understanding  of  the  Recognized  Maritime  Picture  (RMP), 
decision  making  based  on  the  RMP  and  the  efficiency  of  the  RMP  operators’  duties. 

The  RMP  is  a  product  produced  by  the  Regional  Joint  Operation  Centres  (RJOCs).  In  its  common 
form,  it  is  a  map  of  the  Canadian  coastal  waters,  with  contacts,  typically  ships,  marked  on  the  map. 
Each  contact  has  a  set  of  metadata  associated  with  it  which  can  include  (but  is  not  exclusive  to) 
position,  speed,  heading,  ship  name,  hull  number,  threat,  flag,  destination,  origin,  type,  cargo  and  a 
digital  image.  At  worst,  the  metadata  only  consist  of  a  position  (i.e.,  there  is  something  out  there). 
At  best,  the  metadata  consist  of  all  of  the  above.  The  different  degrees  of  metadata  are  due  to  the 
multiple  sources  of  information  that  feed  the  RMP.  These  sources  include  everything  from  radar  to 
surveillance  flights  to  self  reporting  systems  to  voluntary  reports,  each  providing  its  own  subset  of 
data. 

Anomalies  in  the  movement  and  behavior  of  vessels  may  be  of  many  types  and  include,  for 
example: 

•  Unexplained  high  speed:  a  ship  that  is  claiming  to  be  a  normal  merchant  ship  suddenly 
starts  travelling  at  a  high  speed  more  typical  of  a  passenger  ship  or  warship. 

•  Speed  too  slow:  a  Cargo,  Passenger,  or  Ferry  is  observed  going  slowly.  As  these  vessels 
generally  go  as  fast  as  they  safely  can,  it  may  be  an  indicator  of  a  problem. 

•  Loitering:  a  cargo  ship  stops  outside  of  or  far  from  a  harbour,  or  steams  very  slowly,  rather 
than  proceeding  directly  into  port. 
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•  Grab  and  dash  fishing:  a  foreign  fishing  boat  moves  from  the  international  zone  to 
Canadian  waters  (where  it  is  forbidden  from  fishing)  for  a  few  hours  just  before  leaving  for 
its  home  port. 

•  Not  heading  to  port:  a  vessel  is  heading  in  a  direction  where  there  is  no  harbour,  or  is  not 
heading  toward  its  declared  destination.  Cargo  and  Ferry  vessels  always  go  from  one  port 
to  another  port,  and  generally  by  the  shortest  available  route. 

The  purpose  of  the  proposed  study  is  to  explore  the  best  ways  of  alerting  operators  to  such 
anomalies  in  a  non-intrusive  manner.  Given  the  potential  for  many  such  anomalies  during  a 
normal  watch,  it  is  imperative  that  discipline  be  used  in  the  design  of  an  alerting  system  to  ensure 
that  operators  are  not  hindered  in  the  performance  of  their  primary  task  by  nuisance  alerts.  Also,  it 
is  important  that  functionality  is  provided  to  allow  operators  to  define  their  own  criteria  for 
different  alert  priorities  in  different  contexts,  and  that  the  interface  provides  good  situation 
awareness  of  the  different  alert  types.  The  outcome  of  this  work  will  serve  both  the  maritime 
operations  communities,  as  well  the  scientific  communities  in  the  advancement  of  new  methods  for 
the  design  of  non-intrusive  alerting  systems.  This  work  will  be  conducted  by  Humansystems® 
Incorporated. 

Objectives: 

The  goal  of  this  experiment  is  to  explore,  in  the  context  of  the  RMP,  non-intrusive  ways  of 
presenting  to  the  operator  information  concerning  anomalous  behavior  of  vessels.  Because  of  the 
large  number  of  vessel  and  associated  track  data,  it  is  not  currently  possible  for  operators  to  readily 
determine  when  or  where  such  anomalies  occur. 

Overview: 

Six  volunteer  participants  (no  restriction  on  age  or  gender)  will  be  recruited  to  review  a  number  of 
different  design  options  for  a  non-intrusive,  anomaly  alerting  system.  The  participants  will 
individually  do  a  walk  through  of  the  options  which  will  be  presented  as  a  PowerPoint  mock  up  of 
the  screen  interface.  A  questionnaire  will  be  administered  at  the  end  of  each  session,  to  record  the 
volunteers’  subjective  evaluation  of  the  different  design  options  in  terms  of  their  utility  and 
usability.  The  experiment  will  be  undertaken  at  the  MAPLE  laboratory  DRDC  Atlantic  and  will 
require  one  walkthrough  session.  Each  session  will  comprise  approximately  15  minutes  of 
background  briefing  and  a  60-90  minute  walkthrough  session,  which  will  involve  interface 
walkthroughs  and  the  completion  of  a  questionnaire.  This  work  will  be  conducted  by 
Humansystems®  Incorporated. 

Procedures: 

Background  Brie  fing 

Immediately  prior  to  the  walkthrough  session,  participants  will  be  given  an  orientation  briefing  on 
the  overall  study,  its  objectives  and  what  they  will  be  asked  to  do.  At  this  stage  they  will  be  asked 
to  complete  a  consent  form  and  an  information  sheet  about  the  study. 

Walkthrough  session:  Review  of  design  options 

The  goals  of  a  non-intrusive  alerting  system  will  be  described  to  the  participant  and  the  major 
functional  components  of  the  system  will  be  described  at  a  high  level.  Participants  will  then  be 
guided  through  a  PowerPoint  presentation  of  how  a  non-intrusive  alerting  system  interface  would 
look  and  work.  For  each  functional  component  of  the  system,  participants  will  interact  with  an 
animated  PowerPoint  slide  to  simulate  the  actions  of  an  interface.  As  participants  proceed  through 
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the  design  they  will  be  engaged  in  discussion  concerning  how  intuitive  and  easy  the  interface  is  to 
use,  how  it  serves  their  information  needs,  what  information  requirements  are  not  being  met  and 
what  additional  functions  they  would  like  to  see.  It  is  anticipated  that  the  participant  will  be 
presented  with  4-5  design  options  to  review  in  this  manner. 

At  the  completion  of  the  walkthrough  the  participants  will  complete  a  subjective  questionnaire 
documenting  their  evaluation  of  the  different  design  options. 

Participants: 

Approximately  six  Navy,  or  ex-Navy  operators,  will  participate  Navy  operators  will  be  recruited 
through  a  formal  request  through  the  DRDC  Atlantic  Navy  Liaison  Officer.  There  will  be  no 
restriction  on  age  or  gender.  Ex-Navy  participants  will  be  recruited  from  a  list  maintained  by  HSI 
of  ex-Navy  personnel  who  have  indicated  a  prior  willingness  to  be  contacted  as  potential  study 
participants.  .  The  ex-navy  operators  are  males,  Participants  will  be  required  to  self  identify  that 
they  have  normal  colour  vision. 

Equipment  and  Facilities: 

The  apparatus  comprises  a  standard  “Windows”  workstation  with  19”  colour  screen,  a  mouse  and 
keyboard  input,  a  work  surface  to  record  notes,  and  an  ergonomically  designed  operator’s  chair. 

Data  collected: 

The  following  information  will  be  collected  during  each  walkthrough  session: 

Responses  to  the  questionnaires  concerning  the  participant’s  evaluation  of  the  design 
alternatives 

Summary  of  the  participants’  comments  during  free  discussion  with  the  walkthrough 
facilitator 

Experimental  Design/Statistical  Analysis: 

The  small  sample  size  (limited  by  the  availability  of  operational  personnel)  will  likely  have 
insufficient  power  to  warrant  the  use  of  analytical  statistical  procedures  for  estimation  of 
probabilities.  However,  it  should  be  noted  that  the  present  study  is  designed  to  be  an  exploratory 
approach  to  defining  a  preliminary  set  of  good  design  alternatives,  which,  at  some  future  date, 
could  be  evaluated  more  completely  in  a  more  rigorous  experiment. 

Risks  and  Safety  Recommendations: 

This  experiment  offers  minimal  risk  to  the  participant’s  health  and  well-being.  There  is  a  low  risk 
of  eye  fatigue  or  eyestrain,  as  would  be  associated  with  doing  any  visually  intensive  task  (e.g.  web 
searching,  word  processing)  on  computer  display  for  the  period  of  time  used  in  the  walkthrough 
sessions.  This  may  manifest  itself  as  eye  discomfort,  dry  or  itchy  eyes,  or  mild  headache. 

However,  the  duration  of  exposure  to  the  presentation  will  be  short.  Participants  will  be  encouraged 
to  inform  experimenters  if  they  experience  any  discomfort  or  eyestrain,  or  if  they  have  any 
problems  during  the  investigation.  They  may  be  told  to  stop  their  activities  until  problems  or 
conditions  are  resolved.  The  risks  from  participation  in  this  experiment  are  generally  the  same  as 
those  associated  with  the  performance  of  normal  monitoring  of  a  visual  display  that  a  person  might 
do  while  word  processing,  surfing  the  web  or  playing  video  games. 
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Benefits  of  Study: 

Through  their  involvement  in  the  study  participants  will  be  able  to  contribute  to  the  validation, 
development  and  design  of  new  methods  for  providing  information  on  vessel  anomalies,  which  in 
turn  provides  important  human  factors  data  for  the  re-design  and  automation  of  future  systems  to 
represent  the  maritime  picture. 

Informed  Consent: 

Participants  will  be  fully  briefed  on  the  relevant  aspects  of  the  experimental  protocol  and  will  be 
given  a  copy  of  this  protocol  to  review.  They  will  be  required  to  sign  a  voluntary  consent  form, 
indicating  their  willing  informed  consent,  before  being  allowed  to  participate  in  the  experiment.  No 
deception  is  involved. 

Confidentiality: 

Any  personal  or  performance  data  collected  for  each  participant  will  be  available  only  to  the 
experimenters  and  will  be  held  in  the  strictest  confidence.  Participants  will  not  be  identified  by 
name  in  the  data  records;  group  statistics  will  be  used  in  future  presentations  or  publications. 
Individual  participant  data  will  be  coded  anonymously  and  maintained  in  a  computer  file.  The  file 
may  be  accessed  only  by  the  project  team. 

Participant  Debriefing: 

Participants  will  be  permitted  to  ask  any  questions  they  wish  about  the  study  after  they  have 
completed  the  experiment. 

Participant  Stress  Remuneration: 

Participants  will  be  paid  participant  pay  in  the  amount  of  $26.93.  This  assumes  that  the  total  time 
required  will  be  no  more  than  3  hours  per  participant  for  participation  in  this  study  (in  accordance 
with  DRDC  guidelines  and  as  authorized  by  DND  policies15).  The  remuneration  is  calculated  as 
follows: 

3  hours  x  2  (stress  level)  x  $2.50  +$1 1.93  x  1  day  =  $26.93 

Approximate  Time  Involvement: 

All  participants  will  be  required  to  participate  in  one  session  of  approximately  3  hours  with  a  15 
minute  break  in  the  middle. 

Medical  screening: 

No  screening  is  required  for  this  study 

Physician  supervision: 

No  physician  supervision  is  required  for  this  study. 


15  Guide  to  Stress  Compensation  for  Human  Subjects,  R.  Pigeau.  DRDC  Toronto. 
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Voluntary  Consent  Form 


Protocol  Number:  L676 

Research  Project  Title:  Evaluation  of  Interface  Designs  for  Non-Intrusive  Alerts:  Pilot  Study 
Principal  Investigators: 

Dr.  Michael  Matthews,  Humansystems  Incorporated,  Guelph,  Ontario. 

Lora  Bruyn  Martin,  Humansystems  Incorporated,  Guelph,  Ontario 
DRDC  Co-Investigators: 

Ms.  Sharon  McFadden,  DRDC  (Toronto) 

Ms.  Liesa  Lapinski.  DRDC  (Atlantic) 

1.  I, _ 


(Name,  Address,  Phone  number)  hereby  volunteer  to  participate  as  a  participant  in  the  experiment 
entitled  “Evaluation  of  Interface  Designs  for  Non-Intrusive  Alerts:  Pilot  Study”,  the  aim  of  which 
is  to  explore  the  best  ways  of  providing  operators  with  information  on  anomalous  behavior  of 
vessels  in  a  maritime  context.  I  understand  that  I  am  required  to  read  the  attached  protocol  in  its 
entirety.  I  have  had  the  opportunity  to  study  and  discuss  the  attached  protocol  with  the  investigators 
and  I  have  been  informed  to  my  satisfaction  about  the  possible  discomforts  associated  with  these 
tests. 

2.  I  am  aware  that  before  starting  I  will  receive  a  briefing  on  the  aims  and  procedures  for  the 
experiment.  I  will  have  the  opportunity  to  ask  and  receive  answers  to  any  questions  I  may  have.  I 
understand  that  I  am  free  to  refuse  to  participate  and  may  withdraw  my  consent  without  prejudice 
or  hard  feelings  at  any  time.  Should  I  withdraw  my  consent,  my  participation  as  a  participant  will 
cease  immediately.  I  understand  that  the  entire  session  will  last  no  more  than  3  hours. 

3.  I  have  been  told  that  the  principal  risks  associated  with  this  experiment  are  the  possible 
development  of  eyestrain  or  visual  fatigue.  This  may  manifest  itself  as  eye  discomfort,  dry  or  itchy 
eyes,  or  mild  headache.  I  understand  that  the  limited  duration  of  each  experiment  and  rest  periods 
between  experiments  will  mitigate  this  risk.  I  understand  and  accept  this  risk.  I  am  aware  that 
there  are  inherent,  unknown  and  currently  unforeseen  risks  by  DRDCs  Atlantic  and  Toronto  and 
the  Project  Investigators  that  are  associated  with  any  scientific  research  and  that  all  known  risks 
have  been  explained  to  my  satisfaction..  I  have  been  given  examples  of  potential  minor  and  remote 
risks  associated  with  the  experiment  and  consider  these  risks  acceptable  as  well. 

4.  I  agree  to  provide  responses  to  questions  that  are  to  the  best  of  my  knowledge  truthful  and 
complete.  I  have  been  advised  that  the  experimental  data  concerning  me  will  be  treated  as 
confidential  and  not  revealed  to  anyone  other  than  the  investigators  without  my  consent  except  as 
data  unidentified  as  to  source.  I  understand  that  my  name  will  not  be  identified  or  attached  in  any 
manner  to  any  publication  arising  from  this  study. 

5.  I  understand  that  for  my  participation  in  this  research  project,  I  am  entitled  to  stress 
remuneration  in  the  form  of  participant  payment  of  $  26.93.  Stress  remuneration  is  taxable. 
However,  T4A  slips  are  issued  only  for  amounts  in  excess  of  $500.00  remuneration  per  year. 


\lmnansystems 


Non-Intrusive  Alert  System 


Page  A-7 


6.  I  acknowledge  that  I  have  read  this  form  and  I  understand  that  my  consent  is  voluntary  and 
has  been  given  under  circumstances  in  which  I  can  exercise  free  power  of  choice.  I  have  been 
informed  that  I  may,  at  any  time,  revoke  my  consent  and  withdraw  from  the  experiment,  and  that 
the  investigators  may  terminate  my  involvement  in  the  experiment,  regardless  of  my  wishes. 

7.  I  understand  that  by  signing  this  consent  form  I  have  not  waived  any  legal  rights  I  may 
have  as  a  result  of  any  harm  to  me  occasioned  by  my  participation  in  this  research  project  beyond 
all  risks  I  have  assumed. 

8.  [For  Canadian  Forces  (CF)  members  only]  - 1  understand  that  I  am  considered  to  be  on 
duty  for  disciplinary,  administrative  and  Pension  Act  purposes  during  my  participation  in  this 
study.  With  that  said,  this  duty  status  has  no  effect  on  my  right  to  withdraw  for  the  experiment  at 
any  time  I  wish  and  it  is  clear  that  no  action  will  be  taken  against  me  for  exercising  this  right.  As 
well,  in  the  unlikely  event  that  my  participation  in  this  study  results  in  a  medical  condition 
rendering  me  unfit  for  service,  I  may  be  released  from  the  CF  and  my  military  benefits  apply. 


Volunteer’s  Name: 


Signature: _ Date: _ 

Name  of  Witness  to  Signature:  _ 

Signature: _ Date: _ 

Family  Member  or  Contact  Person  (name,  address,  daytime  phone  number  & 
relationship) : _ 

Principal  Investigator: _ 

Signature: _ Date: _ 

FOR  PARTICIPANT  ENQUIRY  IF  REQUIRED: 

Should  I  have  any  questions  or  concerns  regarding  this  project  before,  during,  or  after  participation, 
I  understand  that  I  am  encouraged  to  contact  any  of  the  contacts  below  by  phone  or  e-mail,  to 

Principal  Investigators: 

Dr.  Michael  Matthews,  HumansyVems®  Incorporated  (HSI®),  Guelph,  Ontario  Tel:  (519)  836- 
5911,  email:  mmatthews@humans  vs  .com 

Lora  Bruyn  Martin,  HumansyVems®  Incorporated  (HSI®),  Guelph,  Ontario  Tel:  (519)  836-591 1 
Ex.  303,  email:  lbruyn@humans vs . com 

Co-Investigator: 
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Ms.  Sharon  McFadden,  DRDC  Toronto,  Tel  (416-635-2189),  email:  sharon.mcfadden@drdc- 
rddc.gc.ca 

Chair,  DRDC  Human  Research  Ethics  Committee  (HREC):  Dr.  Jack  P.  Landolt,  phone:  416- 
635-2120,  email:  i ack, landolt@drdc -rddc . gc . ca 

I  understand  that  I  will  be  given  a  copy  of  this  consent  form  so  that  I  may  contact  any  of  the 
above-mentioned  individuals  at  some  time  in  the  future  should  that  be  required. 

Secondary  Use  of  Data:  I  consent/do  not  consent  (delete  as  appropriate)  to  the  use  of  this  study’s 
experimental  data  involving  me  in  unidentified  form  in  future  related  studies  provided  review  and 
approval  have  been  given  by  DRDC  HREC. 


Volunteer’s  Signature 


Date 
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INFORMATION  FOR  PARTICIPANTS 

Title:  Evaluation  of  Interface  Designs  for  Non-Intrusive  Alerts 

Protocol  #L676 

Principal  Investigators:  Dr.  Michael  Matthews  &  Lora  Bruyn  Martin  (HSI®),  Guelph,  Ontario 
Defence  Research  and  Development  Canada  (DRDC)  Co-Investigators: 

Ms.  Sharon  McFadden  (DRDC  Toronto)  &  Ms.  Liesa  Lapinski  (DRDC  Atlantic) 

Background  &  Purpose  of  Study: 

This  applied  research  project  in  the  Maritime  Domain  Awareness  Thrust  is  studying  information 
visualization  and  management  for  enhanced  domain  awareness  in  maritime  security.  The 
DRDC/HSI®  team  wants  to  investigate  the  best  way  to  provide  operators  with  non-intrusive  alerts 
when  certain  forms  of  anomalous  vessel  behavior  occur  to  see  if  the  information  provided  by  way 
of  the  alert  can  help  improve  understanding  of  the  Recognized  Maritime  Picture  (RMP),  decision 
making  based  on  the  RMP  and  the  efficiency  of  the  RMP  operators’  duties.  Given  the  potential  for 
many  such  anomalies  during  a  normal  watch,  it  is  imperative  that  the  design  of  an  alerting  system 
ensures  that  operators  are  not  hindered  in  the  performance  of  their  primary  task  by  nuisance  alerts. 
Also,  it  is  important  that  functionality  is  provided  to  allow  operators  to  define  their  own  criteria  for 
different  alert  priorities  in  different  contexts,  and  that  the  interface  provides  good  situation 
awareness  of  the  different  alert  types.  The  outcome  of  this  work  will  serve  in  the  advancement  of 
new  methods  for  the  design  of  non-intrusive  alerting  systems. 

Procedure: 

Immediately  prior  to  the  walkthrough  session,  you  will  be  given  an  orientation  briefing  on  the 
overall  study,  its  objectives  and  what  you  will  be  asked  to  do.  At  this  stage  you  will  be  asked  to 
complete  a  consent  form. 

The  goals  of  a  non-intrusive  alerting  system  will  be  described  to  you  and  the  major  functional 
components  of  the  system  will  be  described  at  a  high  level.  You  will  then  be  guided  through  a 
PowerPoint  presentation  of  how  a  non-intrusive  alerting  system  interface  would  look  and  work.  For 
each  functional  component  of  the  system,  you  will  interact  with  an  animated  PowerPoint  slide  to 
simulate  the  actions  of  an  interface.  As  you  proceed  through  the  design  you  will  be  engaged  in 
discussion  concerning  the  interface,  how  it  serves  your  information  needs,  what  information 
requirements  are  not  being  met  and  what  additional  functions  you  would  like  to  see.  You  will  be 
presented  with  4-5  design  options  to  review  in  this  manner.  You  will  then  complete  a  subjective 
questionnaire  documenting  your  evaluation  of  the  different  design  options. 

The  session  will  be  approximately  2-3  hours  with  a  15  minute  break  in  the  middle. 
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Annex  B:  Questionnaires  for  SME  Evaluation 


Demographic  Questionnaire 

Instructions:  Please  read  each  question  below  and  write  the  appropriate  response  in  the  answer 
section. 


Question 

Answer 

1 .  Name. 

2.  What  is  your  current  rank? 

3.  What  is  your  current  position? 

4.  How  long  have  you  been  in  that  position? 

5.  Please  list  any  other  related  positions  you  have  had. 
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Usability  and  Usefulness  Questionnaire 


Instructions:  Please  read  each  statement  below  and  circle  the  response  you  feel  is  most 
appropriate  concerning  the  usefulness  of  a  general  non-intrusive  alerting  system. 


Statement 

Strongly 

Disagree 

Disagree 

Undecided 

Agree 

Strongly 

Agree 

1 .  These  alerts  would  enhance  our  knowledge  of 
anomalies 

1 

2 

3 

4 

5 

2.  This  alerting  system  would  be  used  on  a  daily 
basis 

1 

2 

3 

4 

5 

3.  Tasks  can  be  performed  in  a  straightforward 
manner  using  this  alerting  system 

1 

2 

3 

4 

5 

4.  The  thinking  required  to  use  this  alerting 
system  requires  significant  effort 

1 

2 

3 

4 

5 

5.  This  alerting  system  would  be  difficult  to  use 

1 

2 

3 

4 

5 

6.  This  alerting  system  will  improve  my  situation 
awareness 

1 

2 

3 

4 

5 

7.  This  alerting  system  would  make  it  easier  to 
identify  anomalies 

1 

2 

3 

4 

5 

8.  1  would  find  this  alert  system  useful 

1 

2 

3 

4 

5 

9.  1  would  not  ignore  alerts  while  using  this 
technology 

1 

2 

3 

4 

5 

10.  The  Alert  Information  Window  (AIW)  was 
confusing 

1 

2 

3 

4 

5 

11.lt  was  easy  to  learn  how  the  AIW  was 
represented 

1 

2 

3 

4 

5 

12.  The  AIW  had  all  the  necessary  information 

1 

2 

3 

4 

5 

13.  It  was  easy  navigating  between  the  RMP  and 
the  AIW 

1 

2 

3 

4 

5 

14.  1  prefer  clearing  and  deferring  alerts  directly 
from  the  RMP 

1 

2 

3 

4 

5 

15.  1  prefer  using  the  AIW  to  clear  or  defer  alerts 

1 

2 

3 

4 

5 
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Alert  Design  Questionnaire 

Instructions:  Please  read  each  statement  below  and  circle  the  response  you  feel  is  most 
appropriate  concerning  the  usability  of  each  non  intrusive  alert  design. 


Statement  &  Design 

Strongly 

Disagree 

Disagree 

Undecided 

Agree 

Strongly 

Agree 

16.  The  number  of  alerts  were  easy  to  comprehend 

Design  1:  Cumulative  Total  Indicator 

1 

2 

3 

4 

5 

Design  2:  Vertical  Cumulative  Indicator 

1 

2 

3 

4 

5 

Design  3:  Horizontal  Indicator  Bar 

1 

2 

3 

4 

5 

Design  4:  Ticker  &  Fading  Bar 

1 

2 

3 

4 

5 

17.  The  presence  of  an  alert  was  easy  to  recognize 

Design  1:  Cumulative  Total  Indicator 

1 

2 

3 

4 

5 

Design  2:  Vertical  Cumulative  Indicator 

1 

2 

3 

4 

5 

Design  3:  Horizontal  Indicator  Bar 

1 

2 

3 

4 

5 

Design  4:  Ticker  &  Fading  Bar 

1 

2 

3 

4 

5 

18.  The  priorities  of  the  alerts  were  easy  to  comprehend 

Design  1:  Cumulative  Total  Indicator 

1 

2 

3 

4 

5 

Design  2:  Vertical  Cumulative  Indicator 

1 

2 

3 

4 

5 

Design  3:  Horizontal  Indicator  Bar 

1 

2 

3 

4 

5 

Design  4:  Ticker  &  Fading  Bar 

1 

2 

3 

4 

5 

19.  It  was  easy  to  find  the  relevant  anomaly 

Design  1:  Cumulative  Total  Indicator 

1 

2 

3 

4 

5 

Design  2:  Vertical  Cumulative  Indicator 

1 

2 

3 

4 

5 

Design  3:  Horizontal  Indicator  Bar 

1 

2 

3 

4 

5 

Design  4:  Ticker  &  Fading  Bar 

1 

2 

3 

4 

5 

20.  It  was  easy  to  find  information  on  anomalies 

Design  1:  Cumulative  Total  Indicator 

1 

2 

3 

4 

5 

Design  2:  Vertical  Cumulative  Indicator 

1 

2 

3 

4 

5 

Design  3:  Horizontal  Indicator  Bar 

1 

2 

3 

4 

5 

Design  4:  Ticker  &  Fading  Bar 

1 

2 

3 

4 

5 

21.  The  alerting  design  enhanced  my  situation  awareness  of  maritime  anomalies 

Design  1:  Cumulative  Total  Indicator 

1 

2 

3 

4 

5 

Design  2:  Vertical  Cumulative  Indicator 

1 

2 

3 

4 

5 
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Statement  &  Design 

Strongly 

Disagree 

Disagree 

Undecided 

Agree 

Strongly 

Agree 

Design  3:  Horizontal  Indicator  Bar 

1 

2 

3 

4 

5 

Design  4:  Ticker  &  Fading  Bar 

1 

2 

3 

4 

5 

22.  The  appearance  of  the  alerts  is  compatible  with  my  current  interface 

Design  1:  Cumulative  Total  Indicator 

1 

2 

3 

4 

5 

Design  2:  Vertical  Cumulative  Indicator 

1 

2 

3 

4 

5 

Design  3:  Horizontal  Indicator  Bar 

1 

2 

3 

4 

5 

Design  4:  Ticker  &  Fading  Bar 

1 

2 

3 

4 

5 
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Ranking  Questionnaire 

Instructions:  For  the  following  questions,  please  rank  each  of  the  alert  designs  in  order  of 
preference  (where  1  =  most  preferred,  4  =  least  preferred). 


Cumulative  Total  Indicator  Vertical  Cumulative  Horizontal  Indicator  Bar  Ticker  and  Fading  Bar 

Indicator 


23.  In  terms  of  overall  effectiveness  in  bringing  alerts  to  my  attention 

Design  1 :  Cumulative  Total  Indicator 
Design  2:  Vertical  Cumulative  Indicator 
Design  3:  Horizontal  Indicator  Bar 
Design  4:  Ticker  &  Fading  Bar 

24.  In  comparing  the  methods  for  representing  the  different  alert  priorities 

Design  1 :  Cumulative  Total  Indicator 
Design  2:  Vertical  Cumulative  Indicator 
Design  3:  Horizontal  Indicator  Bar 
Design  4:  Ticker  &  Fading  Bar 

25.  In  terms  of  providing  all  of  the  required  information  about  alerts,  without  distracting 
me  from  my  primary  task 

Design  1:  Cumulative  Total  Indicator 

Design  2:  Vertical  Cumulative  Indicator 

Design  3:  Horizontal  Indicator  Bar 

Design  4:  Ticker  &  Fading  Bar 
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Likes  and  Dislikes  Questionnaire 

Instructions:  Please  respond  to  each  alert  design  below  by  indicating  whether  or  not  the 
design  should  be  implemented,  your  likes  and  dislikes,  and  the  design’s  intrusiveness. 


Dislikes: 


26.  Should  Design  1:  Cumulative  Total  Indicator  be  implemented? 

Likes: 


Yes 


1  2 

3 

4  5 

6  7 

Not  at  all  intrusive 

Somewhat  intrusive 

Extremely  intrusive 

27.  Should  Design  2:  Vertical  Cumulative  Indicator  be  implemented?  Yes  No 


1  2 

3 

4  5 

6  7 

Not  at  all  intrusive 

Somewhat  intrusive 

Extremely  intrusive 
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1  2 

3 

4  5 

6  7 

Not  at  all  intrusive 

Somewhat  intrusive 

Extremely  intrusive 

1  2 
Not  at  all  intrusive 


3  4  5 

Somewhat  intrusive 


6  7 

Extremely  intrusive 
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Annex  C:  General  Discussion  Interview 
Questions 

SME  Interview  Questions:  Evaluating  Interface  Designs  for  Non-Intrusive  Alerts 
Things  to  emphasize: 

•  Operators  will  define  priorities  and  select  rules 

•  Show  3  priority  levels  Rory  created 

•  Show  rule  selection  window 

General: 

•  At  desk  vs.  away  from  desk 

•  Looking  at  screen  vs.  not  looking  at  screen 

•  Frequency  of  alerts  (by  priority) 

•  Should  priority  1  alerts  be  intrusive? 

•  Ignoring  alerts 

•  Did  you  like  the  way  the  priorities  were  indicated? 

Interface  design: 

•  Alert  colour  scheme  (e.g.,  red,  orange,  yellow,  grey) 

•  Font  (e.g.,  size,  colour) 

•  Was  the  terminology  used  familiar,  clear  and  understandable? 

•  Alert  Information  Window  -  useful  and  comprehensiveness? 

•  Navigation  -  can  you  find  the  relevant  information? 

•  Intuitive  versus  not  intuitive 

•  Intrusiveness  of  alerts 

•  Should  there  be  auditory/tactile  alerts  in  tandem  or  separately? 

•  Flexibility  in  the  location  of  ticker  and  horizontal  indicator  bar 

•  Would  operators  want  RMP  centred  on  contact? 

•  Would  you  want  more  information  in  the  pop  up  box  in  the  RMP? 

Context  of  use: 

•  Ability  to  return  to  primary  task 

•  Potential  problems  from  using  the  non-intrusive  alert  designs  in  practice? 

o  During  an  increased  workload? 
o  When  you  are  not  at  your  computer  screen? 
o  At  shift  change  over?  Beginning  or  end  of  shift 
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Acronyms 


ABAIS 

Affect  and  Belief  Adaptive  Interface  System 

ACT 

Alarm  Cleanup  Toolbox 

AIS 

Automatic  Identification  System 

AIW 

Alert  Information  Window 

ANOVA 

Analysis  of  Variance 

ARO 

Assistant  Reactor  Operator 

ATCS 

Air  Traffic  Control  Specialists 

ATM 

Air  Traffic  Management 

AVMS 

AIS  Vessel  Monitoring  System 

CAPTA 

Cognitive,  Affective,  Personality  Task  Analysis 

CCS 

Combat  Control  System 

CDTI 

Cockpit  Displays  of  Traffic  Information 

CHDB 

Contact  History  Database 

CHEX 

Change  History  Explicit 

CISTI 

Canada  Institute  for  Scientific  and  Technical  Information 

CONOPS 

Concept  of  Operations 

COTS 

Commercial-Off-The-Shelf 

CS 

Combat  System 

DCS 

Distributed  Control  System 

DRDC 

Defence  Research  and  Development  Canada 

DSS 

Decision  Support  System 

ELINT 

Electronic  Intelligence 

ETA 

Estimated  Time  of  Arrival 

FCR 

Fire-Control  Radar 

FFC 

Bio-Fuelled  District  Heating  Plant 

GCCS-M 

Global  Command  and  Control  System-Maritime 

GSFC 

Goddard  Space  Flight  Center 

GUI 

Graphic  User  Interface 

HAIL 

Human  Alerting  and  Interruption  Logistics 

HCI 

Human  Computer  Interaction 

HF 

Human  Factors 

HFSWR 

High  Frequency  Surface  Wave  Radar 

HMIAS 

Hazard  Monitor  and  Intelligent  Alerting  System 

IAMS 

Intelligent  Alarm  Management  System 

ID 

Identification 
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IDS 

Intrusion  Detection  Systems 

IOP 

Input  Open 

IP 

Internet  Protocol 

IRC 

Interruption,  Reaction  and  Comprehension 

IS 

Identification  Supervisor 

LCCI 

Large  Complex  Critical  Infrastructure 

LCL 

Lower  Control  Limits 

MARLANT 

Maritime  Forces  Atlantic 

MISR 

Maritime  Intelligence,  Surveillance,  and  Reconnaissance 

MMI 

Man-Machine  Interface 

MMSI 

Maritime  Mobile  Service  Identity 

MPA 

Maritime  Patrol  Aircraft 

MSOCC 

Multisatellite  Operations  Control  Center 

NAS 

National  Airspace  System 

NASA 

National  Aeronautics  and  Space  Administration 

NATO 

North  Atlantic  Treaty  Organization 

NRC 

Nuclear  Regulatory  Commission 

OFMspert 

Operator  Function  Model  Expert  System 

PAL 

Provincial  Airlines 

PDA 

Personal  Digital  Assistant 

PL 

Platoon  Leaders 

R&D 

Research  and  Development 

RJOC 

Regional  Joint  Operations  Center 

RMP 

Recognized  Maritime  Picture 

RO 

Reactor  Operator 

RSVP 

Rapid  Serial  Visual  Presentation 

SHIELD 

System  to  Help  Identify  and  Empower  Leader  Decisions 

SME 

Subject  Matter  Expert 

SOP 

Standard  Operating  Procedure 

TA 

Technical  Authority 

VDU 

Visual  Display  Unit 

VOI 

Vessel  of  Interest 

UAV 

Unmanned  Aerial  Vehicle 

UCL 

Upper  Control  Limits 
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13.  ABSTRACT  (A  brief  and  factual  summary  of  the  document.  It  may  also  appear  elsewhere  in  the  body  of  the  document  itself.  It  is  highly  desirable  that 
the  abstract  of  classified  documents  be  unclassified.  Each  paragraph  of  the  abstract  shall  begin  with  an  indication  of  the  security  classification  of  the 
information  in  the  paragraph  (unless  the  document  itself  is  unclassified)  represented  as  (S),  (C),  (R),  or  (U).  It  is  not  necessary  to  include  here  abstracts  in 
both  official  languages  unless  the  text  is  bilingual.) 

This  project  involves  the  investigation  of  best  practices  for  the  development  of  design  concepts  for  a 
visualization  aid,  specifically  an  alerting  system,  which  would  increase  the  RMP  operators’  awareness 
and  understanding  of  maritime  anomalies  in  the  RMP  (e.g.  vessel  not  heading  to  port,  grab  and  dash 
fishing,  etc.).  Such  an  alerting  system,  however,  must  make  operators  aware  of  anomalies  that  may  be 
present  without  impacting  on  the  performance  of  their  primary  tasks. 

The  objectives  of  this  project  were  (i)  to  identify  and  analyse  available  literature  relevant  to  non-intrusive 
alert  systems,  (ii)  develop  design  concepts  for  a  non-intrusive  alerting  interface  to  be  used  in  GCCS-M 
and  (iii)  obtain  feedback  from  Navy  Subject  Matter  Experts  (SMEs)  on  the  suitability  of  the  design 
options. 

The  results  of  the  literature  review  suggest  that  there  is  a  lack  of  a  unified  design  approach  and  associated 
recommendations  for  non-intrusive  alerting  contexts.  Furthermore,  there  was  no  single  paper  that 
definitively  addressed  the  issue  of  how  to  design  a  non-intrusive  alerting  system.  However,  we  were  able 
to  extract  relevant  concepts  from  the  literature  relating  to  alert/alarm  design  in  general.  These  concepts, 
combined  with  general  human  factors  principles,  provided  direction  for  a  number  of  design  concepts 
which  were  then  reviewed  and  evaluated  by  subject  matter  experts. 

Future  design  efforts  should  work  toward  developing  an  alert  system  interface  design  in  accordance  with 
the  design  principles  listed  above,  once  these  design  requirements  have  been  validated. 

Le  projet  comprend  T etude  des  meilleures  pratiques  applicables  a  la  definition  de  concepts  pour  un 
systeme  d’aide  a  la  visualisation,  en  T occurrence  un  systeme  d’alerte,  qui  aiderait  les  operateurs  du  TSM 
a  mieux  connaitre  et  comprendre  les  anomalies  maritimes  indiquees  dans  le  TSM  (deroutement  d’un 
navire,  braconnage  maritime,  etc.).  Un  tel  systeme  d’alerte  doit  toutefois  permettre  aux  operateurs  d’etre 
informes  des  anomalies  eventuelles  sans  pour  autant  entraver  1’ execution  de  leurs  taches  principales. 

Le  projet  visait  les  objectifs  suivants  :  (i)  identifier  et  analyser  la  documentation  disponible  sur  les 
systemes  d’alerte  non  intrusive,  (ii)  elaborer  des  concepts  pour  une  interface  d’alerte  non  intrusive  a 
utiliser  dans  le  GCCS-M  et  (iii)  obtenir  la  retroaction  des  experts  de  la  Marine  sur  la  valeur  des  options  de 
conception. 

L’examen  de  la  documentation  a  revele  1’ absence  d’une  approche  de  conception  unifiee  et  de 
recommandations  associees  dans  le  contexte  d’alertes  non  intrusives.  En  outre,  aucun  document  n’offrait 
de  solution  definitive  au  probleme  de  la  conception  d’un  systeme  d’alerte  non  intrusive.  Toutefois,  on  a 
pu  extraire  de  la  documentation  des  concepts  pertinents  pour  la  conception  de  systemes  d’alerte  et 
d’alarme  en  general.  Ces  concepts,  associes  a  des  principes  generaux  touchant  les  facteurs  humains,  ont 
foumi  des  orientations  pour  la  definition  d’un  certain  nombre  de  concepts  qui  ont  ensuite  ete  examines  et 
evalues. 

Les  recherches  futures  devraient  viser  a  definir  une  conception  d’ interface  de  systeme  d’alerte 
conformement  aux  principes  de  conception  presentes  ci-dessus,  une  fois  que  ces  exigences  de  conception 
auront  ete  validees. 
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Thesaurus  of  Engineering  and  Scientific  Terms  (TEST)  and  that  thesaurus  identified.  If  it  is  not  possible  to  select  indexing  terms  which  are  Unclassified, 
the  classification  of  each  should  be  indicated  as  with  the  title.) 
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